Identifying Named Entities of Adverse Drug Reaction with Adversarial Transfer Learning
Han Pu1,2(),Zhong Yule1,Lu Haojie1,Ma Shiwen1
1School of Management, Nanjing University of Posts & Telecommunications, Nanjing 210003, China 2Jiangsu Provincial Key Laboratory of Data Engineering and Knowledge Service, Nanjing 210023, China
[Objective] This paper proposes an entity recognition model for adverse drug reactions based on adversarial transfer learning, ATL-BCA, aiming to address the problem of non-standard entity representations and insignificant boundaries in online health communities. [Methods] Firstly, we generated the external semantic feature vectors fused with the online medical domain knowledge with Word2Vec. Secondly, based on the transfer learning, we utilized the shared and private BiLSTM to extract the shared boundary information and private features for entity recognition and word segmentation tasks. Next, we used the multi-head attention mechanism to capture the overall sentence dependency and used adversarial training to filter the private information of the word segmentation task. This helped us eliminate the influence of redundant features on the entity recognition task. Finally, we predicted the label sequence results with the help of CRF constraints. [Results] We used a self-constructed social media adverse drug reaction dataset to examine the proposed model with. The F1 value of the new model reached 91.35%, which is 5.28% and 2.98% higher than Word2Vec-BiLSTM-CRF and BERT-BiLSTM-CRF. [Limitations] We only retrieved the experimental data from Sanjiu Health & Medicine Site, the scale of the constructed dataset is relatively small. [Conclusions] The ATL-BCA model fully utilizes the shared boundary information between entity recognition and word segmentation tasks. It also filters the private features of the word segmentation tasks, effectively improving the entity recognition performance of adverse drug reactions in online health communities.
韩普, 仲雨乐, 陆豪杰, 马诗雯. 基于对抗性迁移学习的药品不良反应实体识别研究*[J]. 数据分析与知识发现, 2023, 7(3): 131-141.
Han Pu, Zhong Yule, Lu Haojie, Ma Shiwen. Identifying Named Entities of Adverse Drug Reaction with Adversarial Transfer Learning. Data Analysis and Knowledge Discovery, 2023, 7(3): 131-141.
Nikfarjam A, Sarker A, O’Connor K, et al. Pharmacovigilance from Social Media: Mining Adverse Drug Reaction Mentions Using Sequence Labeling with Word Embedding Cluster Features[J]. Journal of the American Medical Informatics Association, 2015, 22(3): 671-681.
doi: 10.1093/jamia/ocu041
pmid: 25755127
( Liu Pengcheng, Wang Jiayu, Chen Jinmin, et al. Pharmacovigilance Laws in Europe and America and Their Implications for China[J]. Chinese Journal of Pharmacovigilance, 2020, 17(12): 877-882.)
( Zhang Yafei, Yu Qi, Wang Yuxin, et al. Establishment of Knowledge Discovery System for Potential Adverse Reactions of Drugs and Indications for Their Use in Drug Forum[J]. Chinese Journal of Medical Library and Information Science, 2020, 29(7): 38-43.)
( Fan Xinyue, Cui Lei. Using Text Mining to Discover Drug Side Effects: Case Study of PubMed[J]. Data Analysis and Knowledge Discovery, 2018, 2(3): 79-86.)
[5]
Leaman R, Wojtulewicz L, Sullivan R, et al. Towards Internet-Age Pharmacovigilance: Extracting Adverse Drug Reactions from User Posts to Health-Related Social Networks[C]// Proceedings of the 2010 Workshop on Biomedical Natural Language Processing. ACM, 2010: 117-125.
( Liu Liu, Wang Dongbo. A Review on Named Entity Recognition[J]. Journal of the China Society for Scientific and Technical Information, 2018, 37(3): 329-340.)
( Yang Piao, Dong Wenyong. Chinese Named Entity Recognition Method Based on BERT Embedding[J]. Computer Engineering, 2020, 46(4): 40-45.)
[9]
Luo W C, Yang F. An Empirical Study of Automatic Chinese Word Segmentation for Spoken Language Understanding and Named Entity Recognition[C]// Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics:Human Language Technologies. ACL, 2016: 238-248.
[10]
Peng N Y, Dredze M. Improving Named Entity Recognition for Chinese Social Media with Word Segmentation Representation Learning[OL]. arXiv Preprint, arXiv: 1603.00786.
[11]
Denton E, Chintala S, Szlam A, et al. Deep Generative Image Models Using a Laplacian Pyramid of Adversarial Networks[OL]. arXiv Preprint, arXiv: 1506.05751.
[12]
Tzeng E, Hoffman J, Saenko K, et al. Adversarial Discriminative Domain Adaptation[C]// Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. 2017: 2962-2971.
[13]
Gurulingappa H, Mateen-Rajput A, Toldo L. Extraction of Potential Adverse Drug Events from Medical Case Reports[J]. Journal of Biomedical Semantics, 2012, 3(1): 15.
doi: 10.1186/2041-1480-3-15
pmid: 23256479
[14]
Xu J, Lee H J, Ji Z C, et al. UTH_CCB System for Adverse Drug Reaction Extraction from Drug Labels at TAC-ADR 2017[C]// Proceedings of the 2017 Text Analysis Conference. 2017.
[15]
Narayanan S, Mannam K, Rajan S P, et al. Evaluation of Transfer Learning for Adverse Drug Event and Medication Entity Extraction[C]// Proceedings of the 3rd Clinical Natural Language Processing Workshop. 2020: 55-64.
[16]
Florez E, Precioso F, Riveill M, et al. Named Entity Recognition Using Neural Networks for Clinical Notes[C]// Proceedings of the 2018 International Workshop on Medication and Adverse Drug Event Detection. PMLR, 2018: 7-15.
[17]
Kang N, Singh B, Bui C, et al. Knowledge-Based Extraction of Adverse Drug Events from Biomedical Text[J]. BMC Bioinformatics, 2014, 15: 64.
doi: 10.1186/1471-2105-15-64
pmid: 24593054
[18]
Dandala B, Joopudi V, Devarakonda M. Adverse Drug Events Detection in Clinical Notes by Jointly Modeling Entities and Relations Using Neural Networks[J]. Drug Safety, 2019, 42(1): 135-146.
doi: 10.1007/s40264-018-0764-x
pmid: 30649738
[19]
Chen Y, Zhou C, Li T, et al. Named Entity Recognition from Chinese Adverse Drug Event Reports with Lexical Feature Based BiLSTM-CRF and Tri-Training[J]. Journal of Biomedical Informatics, 2019, 96: 103252.
doi: 10.1016/j.jbi.2019.103252
[20]
Christopoulou F, Tran T T, Sahu S K, et al. Adverse Drug Events and Medication Relation Extraction in Electronic Health Records with Ensemble Deep Learning Methods[J]. Journal of the American Medical Informatics Association, 2020, 27(1): 39-46.
doi: 10.1093/jamia/ocz101
pmid: 31390003
[21]
Wu H, Ji J T, Tian H M, et al. Chinese-Named Entity Recognition from Adverse Drug Event Records: Radical Embedding-Combined Dynamic Embedding-Based BERT in a Bidirectional Long Short-Term Conditional Random Field (Bi-LSTM-CRF) Model[J]. JMIR Medical Informatics, 2021, 9(12): e26407.
doi: 10.2196/26407
[22]
Cocos A, Fiks A G, Masino A J. Deep Learning for Pharmacovigilance: Recurrent Neural Network Architectures for Labeling Adverse Drug Reactions in Twitter Posts[J]. Journal of the American Medical Informatics Association, 2017, 24(4): 813-821.
doi: 10.1093/jamia/ocw180
pmid: 28339747
( Wei Wei, Zheng Du. The Study of Adverse Drug Reaction Signal Extraction Framework Based on the Integrated Statistical Learning and Semantic Filter[J]. Library and Information Service, 2018, 62(5): 115-124.)
doi: 10.13266/j.issn.0252-3116.2018.05.013
( Mu Dongmei, Ju Yuanhong, Dai Wenhao, et al. Knowledge Discovery Strategy and Model of Virtual Health Community Text Data[J]. Library and Information Service, 2018, 62(5): 125-131.)
doi: 10.13266/j.issn.0252-3116.2018.05.014
[25]
Metke-Jimenez A, Karimi S. Concept Extraction to Identify Adverse Drug Reactions in Medical Forums: A Comparison of Algorithms[OL]. arXiv Preprint, arXiv: 1504.06936.
[26]
Yates A, Goharian N, Frieder O. Extracting Adverse Drug Reactions from Social Media[C]// Proceedings of the 29th AAAI Conference on Artificial Intelligence. ACM, 2015: 2460-2467.
[27]
Tang B Z, Hu J L, Wang X L, et al. Recognizing Continuous and Discontinuous Adverse Drug Reaction Mentions from Social Media Using LSTM-CRF[J]. Wireless Communications and Mobile Computing, 2018, 2018: 1-8.
[28]
Stanovsky G, Gruhl D, Mendes P. Recognizing Mentions of Adverse Drug Reaction in Social Media Using Knowledge-Infused Recurrent Models[C]// Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics. 2017: 142-151.
[29]
Lehmann J, Isele R, Jakob M, et al. DBpedia—a Large-Scale, Multilingual Knowledge Base Extracted from Wikipedia[J]. Semantic Web, 2015, 6(2): 167-195.
doi: 10.3233/SW-140134
[30]
Asghari M, Sierra-Sosa D, Elmaghraby A S. BINER: A Low-Cost Biomedical Named Entity Recognition[J]. Information Sciences, 2022, 602: 184-200.
doi: 10.1016/j.ins.2022.04.037
( Zhu Xiaoxiao, Yang Zunqi, Liu Jing. Construction of an Adverse Drug Reaction Extraction Model Based on Bi-LSTM and CRF[J]. Data Analysis and Knowledge Discovery, 2019, 3(2): 90-97.)
[32]
Ruder S, Peters M E, Swayamdipta S, et al. Transfer Learning in Natural Language Processing[C]// Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics. 2019: 15-18.
[33]
Yadav S, Ekbal A, Saha S, et al. A Unified Multi-Task Adversarial Learning Framework for Pharmacovigilance Mining[C]// Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. 2019: 5234-5245.
( Dong Zhe, Shao Ruoqi, Chen Yuliang, et al. Named Entity Recognition in Food Field Based on BERT and Adversarial Training[J]. Computer Science, 2021, 48(5): 247-253.)
doi: 10.11896/jsjkx.200800181
[36]
Cao P F, Chen Y B, Liu K, et al. Adversarial Transfer Learning for Chinese Named Entity Recognition with Self-Attention Mechanism[C]// Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. 2018: 182-192.
[37]
Bekoulis G, Deleu J, Demeester T, et al. Adversarial Training for Multi-Context Joint Entity and Relation Extraction[OL]. arXiv Preprint, arXiv: 1808.06876.
[38]
Zhang T, Lin H, Ren Y, et al. Identifying Adverse Drug Reaction Entities from Social Media with Adversarial Transfer Learning Model[J]. Neurocomputing, 2021, 453: 254-262.
doi: 10.1016/j.neucom.2021.05.007
[39]
Kang K M, Tian S W, Yu L. Named Entity Recognition of Local Adverse Drug Reactions in Xinjiang Based on Transfer Learning[J]. Journal of Intelligent & Fuzzy Systems, 2021, 40(5): 8899-8914.
[40]
Cornegruta S, Bakewell R, Withey S, et al. Modelling Radiological Language with Bidirectional Long Short-Term Memory Networks[C]// Proceedings of the 7th International Workshop on Health Text Mining and Information Analysis. 2016: 17-27.
( Zhu Zhangli, Rao Yuan, Wu Yuan, et al. Research Progress of Attention Mechanism in Deep Learning[J]. Journal of Chinese Information Processing, 2019, 33(6): 1-11.)
( Shi Lei, Wang Yi, Cheng Ying, et al. Review of Attention Mechanism in Natural Language Processing[J]. Data Analysis and Knowledge Discovery, 2020, 4(5): 1-14.)
[43]
Vaswani A, Shazeer N, Parmar N, et al. Attention is All You Need[C]// Proceedings of the 31st International Conference on Neural Information Processing Systems. ACM, 2017: 6000-6010.
[44]
Sutton C, McCallum A. An Introduction to Conditional Random Fields for Relational Learning[A]// Koller D. Introduction to Statistical Relational Learning[M]. Cambridge: MIT Press, 2006.
[45]
Carlezon W A, Béguin C, Knoll A T, et al. Kappa-Opioid Ligands in the Study and Treatment of Mood Disorders[J]. Pharmacology & Therapeutics, 2009, 123(3): 334-343.
[46]
Subramanyam K K, Sangeetha S. Deep Contextualized Medical Concept Normalization in Social Media Text[J]. Procedia Computer Science, 2020, 171: 1353-1362.
doi: 10.1016/j.procs.2020.04.145
