1School of Economics and Management, Beijing Jiaotong University, Beijing 100044, China 2National Clinical Medical Research Center for Nervous System Diseases, Beijing Tiantan Hospital Affiliated to Capital Medical University, Beijing 100050, China 3Community Health Service Center, Beijing Jiaotong University, Beijing 100044, China
[Objective] This study tries to address the issues of polysemy and incomplete words facing entity recognition for Chinese Electronic Medical Records (EMR). [Methods] We constructed a deep learning model RoBERTa-WWM-BiLSTM-CRF to improve the named entity recognition of Chinese EMR. We conducted four rounds of experiments to compare their impacts on entity recognition. [Results] The highest F1 value of the new model reached 0.8908. [Limitations] The experiment data set is small, and the entity recognition results of some departments was not very impressive. For example, the F1 value of respiratory department was only 0.8111. [Conclusions] The RoBERTa-WWM-BiLSTM-CRF model could effectively conduct named entity recognition for Chinese electronic medical records.
Grishman R, Sundheim B. Message Understanding Conference 6: A Brief History[C]// Proceedings of the 16th Conference on Computational Linguistics-Volume 1. 1996: 446-471.
[2]
Jang H J, Cho K O. Applications of Deep Learning for the Analysis of Medical Data[J]. Archives of Pharmacal Research, 2019, 42(6):492-504.
doi: 10.1007/s12272-019-01162-9
[3]
Ubbens J R, Stavness I. Deep Plant Phenomics: A Deep Learning Platform for Complex Plant Phenotyping Tasks[J]. Frontiers in Plant Science, 2017, 8:1190.
doi: 10.3389/fpls.2017.01190
[4]
Belle A, Thiagarajan R, Soroushmehr S M R, et al. Big Data Analytics in Healthcare[J]. BioMed Research International, 2015: 1-16.
[5]
Shen L, Li Q, Wang W, et al. Treatment Patterns and Direct Medical Costs of Metastatic Colorectal Cancer Patients: A Retrospective Study of Electronic Medical Records from Urban China[J]. Journal of Medical Economics, 2020, 23(5):456-463.
doi: 10.1080/13696998.2020.1717500
pmid: 31950863
[6]
Friedman C, Kra P, Rzhetsky A. Two Biomedical Sublanguages: A Description Based on the Theories of Zellig Harris[J]. Journal of Biomedical Informatics, 2002, 35(4):222-235.
pmid: 12755517
( Yang Jinfeng, Yu Qiubin, Guan Yi, et al. An Overview of Research on Electronic Medical Record Oriented Named Entity Recognition and Entity Relation Extraction[J]. Acta Automatica Sinica, 2014, 40(8):1537-1562.)
[8]
Ganslandt T, Prokosch H U. Perspectives for Medical Informatics[J]. Methods of Information in Medicine, 2009, 48(1):38-44.
pmid: 19151882
[9]
Greenes R A, Shortliffe E H. Medical Informatics: An Emerging Academic Discipline and Institutional Priority[J]. JAMA: The Journal of the American Medical Association, 1990, 263(8):1114-1120.
doi: 10.1001/jama.1990.03440080092030
( Li Chunying, Zhu Lan, Lang Jinghe, et al. Exploratory and Confirmatory Factor Analyses for Testing Validity and Reliability of the Chinese Language Questionnaire for Urinary Incontinence Diagnosis[J]. Chinese Journal of Obstetrics and Gynecology, 2016, 51(5):357-360.)
[11]
Kim H K. Health Informatics: A Telehealth User- Friendly Design Monitoring Approach[J]. International Journal of Control and Automation, 2017, 10(12):89-98.
[12]
Gardner R M, Overhage J M, Steen E B, et al. Core Content for the Subspecialty of Clinical Informatics[J]. Journal of the American Medical Informatics Association, 2009, 16(2):153-157.
doi: 10.1197/jamia.M3045
pmid: 19074296
[13]
Frankovich J, Longhurst C A, Sutherland S M. Evidence-Based Medicine in the EMR Era[J]. The New England Journal of Medicine, 2011, 365(19):1758-1759.
doi: 10.1056/NEJMp1108726
pmid: 22047518
[14]
Fowler S A, Yaeger L H, Yu F, et al. Electronic Health Record: Integrating Evidence-Based Information at the Point of Clinical Decision Making[J]. Journal of the Medical Library Association, 2014, 102(1):52-55.
doi: 10.3163/1536-5050.102.1.010
pmid: 24415920
[15]
Eysenbach G. Consumer Health Informatics[J]. British Medical Journal (Clinical researched.), 2000, 320(7251):1713.
[16]
Alpay L, Verhoef J, Xie B, et al. Current Challenge in Consumer Health Informatics: Bridging the Gap Between Access to Information and Information Understanding[J]. Biomedical Informatics Insights, 2009, 2(1):1-10.
[17]
Wiesner M, Pfeifer D. Health Recommender Systems: Concepts, Requirements, Technical Basics and Challenges[J]. International Journal of Environmental Research and Public Health, 2014, 11(3):2580-2607.
doi: 10.3390/ijerph110302580
pmid: 24595212
[18]
Vapnik V N, Lerner A Y. Recognition of Patterns with Help of Generalized Portraits[J]. Avtomatika i Telemekhanika, 1963, 24(6):774-780.
[19]
Zhou G, Shen D, Zhang J, et al. Recognition of Protein/Gene Names from Text Using an Ensemble of Classifiers[J]. BMC Bioinformatics, 2005, 6(S1):S7.
[20]
Lafferty J D, McCallum A, Pereira F C N. Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data[C]// Proceedings of the 18th International Conference on Machine Learning. 2001: 282-289.
[21]
Jonnalagadda S, Cohen T, Wu S, et al. Enhancing Clinical Concept Extraction with Distributional Semantics[J]. Journal of Biomedical Informatics, 2012, 45(1):129-140.
doi: 10.1016/j.jbi.2011.10.007
pmid: 22085698
[22]
Jiang M, Chen Y K, Liu M, et al. A Study of Machine-Learning-Based Approaches to Extract Clinical Entities and Their Assertions from Discharge Summaries[J]. Journal of the American Medical Informatics Association, 2011, 18(5):601-606.
doi: 10.1136/amiajnl-2011-000163
pmid: 21508414
[23]
Cocos A, Fiks A G, Masino A J. Deep Learning for Pharmacovigilance: Recurrent Neural Network Architectures for Labeling Adverse Drug Reactions in Twitter Posts[J]. Journal of the American Medical Informatics Association, 2017, 24(4):813-821.
doi: 10.1093/jamia/ocw180
pmid: 28339747
( Zhang Fan, Wang Min. Medical Text Entities Recognition Method Base on Deep Learning[J]. Computing Technology and Automation, 2017, 36(1):123-127.)
[25]
LeCun Y, Bengio Y. Convolutional Networks for Images, Speech, and Time Series[A]// The Handbook of Brain Theory and Neural Networks[M]. 1998: 255-258.
[26]
Schmidhuber J. Deep Learning in Neural Networks: An Overview[J]. Neural Networks, 2015, 61:85-117.
pmid: 25462637
[27]
Sundermeyer M, Schlüter R, Ney H. LSTM Neural Networks for Language Modeling[C]// Proceedings of Interspeech 2012. 2012:601-608.
[28]
Mikolov T, Chen K, Corrado G, et al. Efficient Estimation of Word Representations in Vector Space[OL]. arXiv Preprint, arXiv:1301.3781.
[29]
Devlin J, Chang M W, Lee K, et al. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding[OL]. arXiv Preprint, arXiv: 1810.04805.
[30]
Wu Y, Jiang M, Xu J, et al. Clinical Named Entity Recognition Using Deep Learning Models[J]. AMIA Annual Symposium Proceedings AMIA Symposium, 2017, 2017:1812-1819.
[31]
Hochreiter S, Schmidhuber J. Long Short-Term Memory[J]. Neural Computation, 1997, 9(8):1735-1780.
pmid: 9377276
[32]
Fukada T, Schuster M, Sagisaka Y. Phoneme Boundary Estimation Using Bidirectional Recurrent Neural Networks and Its Applications[J]. Systems and Computers in Japan, 1999, 30(4):20-30.
[33]
Habibi M, Weber L, Neves M, et al. Deep Learning with Word Embeddings Improves Biomedical Named Entity Recognition[J]. Bioinformatics, 2017, 33(14):i37-i48.
doi: 10.1093/bioinformatics/btx228
[34]
Wang X, Zhang Y, Ren X, et al. Cross-Type Biomedical Named Entity Recognition with Deep Multi-Task Learning[J]. Bioinformatics, 2019, 35(10):1745-1752.
doi: 10.1093/bioinformatics/bty869
[35]
Topaz M, Murga L, Gaddis K M, et al. Mining Fall-Related Information in Clinical Notes: Comparison of Rule-Based and Novel Word Embedding-Based Machine Learning Approaches[J]. Journal of Biomedical Informatics, 2019, 90:103103.
doi: 10.1016/j.jbi.2019.103103
( Yang Yanling, Li Yan, Zhong Xinyu, et al. Named Entity Recognition of TCM Medical Records Based on BiLSTM-CRF[J]. Information on Traditional Chinese Medicine, 2021, 38(11):15-21.)
[37]
Kim Y M, Lee T H. Korean Clinical Entity Recognition from Diagnosis Text Using BERT[J]. BMC Medical Informatics and Decision Making, 2020, 20(suppl 7):242.
doi: 10.1186/s12911-020-01241-8
[38]
Lee J, Yoon W, Kim S, et al. BioBERT: A Pre-Trained Biomedical Language Representation Model for Biomedical Text Mining[J]. Bioinformatics, 2020, 36(4):1234-1240.
[39]
Zhang W, Jiang S, Zhao S, et al. A BERT-BiLSTM-CRF Model for Chinese Electronic Medical Records Named Entity Recognition[C]// Proceedings of the 12th International Conference on Intelligent Computation Technology and Automation (ICICTA). 2019.
[40]
Cho H, Lee H. Biomedical Named Entity Recognition Using Deep Neural Networks with Contextual Information[J]. BMC Bioinformatics, 2019, 20(1):735.
doi: 10.1186/s12859-019-3321-4
[41]
Cui Y M, Che W X, Liu T, et al. Pre-Training with Whole Word Masking for Chinese BERT[J]. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2021, 29:3504-3514.
doi: 10.1109/TASLP.2021.3124365
[42]
Zhu X X, Li L X, Liu J, et al. Captioning Transformer with Stacked Attention Modules[J]. Applied Sciences, 2018, 8(5):739.
doi: 10.3390/app8050739
[43]
Liu Y, Ott M, Goyal N, et al. RoBERTa: A Robustly Optimized BERT Pretraining Approach[OL]. arXiv Preprint, arXiv: 1907.11692.
[44]
Ma X L, Tao Z M, Wang Y H, et al. Long Short-Term Memory Neural Network for Traffic Speed Prediction Using Remote Microwave Sensor Data[J]. Transportation Research Part C: Emerging Technologies, 2015, 54:187-197.
doi: 10.1016/j.trc.2015.03.014
[45]
Greff K, Srivastava R K, Koutnik J, et al. LSTM: A Search Space Odyssey[J]. IEEE Transactions on Neural Networks and Learning Systems, 2017, 28(10):2222-2232.
doi: 10.1109/TNNLS.2016.2582924
[46]
Cornegruta S, Bakewell R, Withey S, et al. Modelling Radiological Language with Bidirectional Long Short-Term Memory Networks[C]// Proceedings of the 7th International Workshop on Health Text Mining and Information Analysis. Austin: Association for Computational Linguistics, 2016: 17-27.
( Ma Mengcheng, Yang Qingwen, Askar · Hamdulla, et al. Chinese Named Entity Classification Based on Word Vector and Conditional Random Fields[J]. Computer Engineering and Design, 2020, 41(9):2515-2522.)
( Bai Bing, Hou Xia, Shi Song. Named Entity Recognition Method Based on CRF and Bi-LSTM[J]. Journal of Beijing Information Science & Technology University, 2018, 33(6):27-33.)
[49]
Zweig G, Nguyen P, van Compernolle D, et al. Speech Recognitionwith Segmental Conditional Random Fields: A Summary of the JHU CLSP 2010 Summer Workshop[C]// Proceedings of 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 2011: 5044-5047.
[50]
Paszke A, Gross S, Chintala S, et al. Automatic Differentiation in PyTorch[C]// Proceedings of the 31st Conference on Neural Information Processing Systems. 2017.
