Extracting Relationship Among Military Domains with BERT and Relation Position Features
Ma Jiangwei1,Lv Xueqiang1,You Xindong1(),Xiao Gang2,Han Junmei2
1Beijing Key Laboratory of Internet Culture and Digital Dissemination Research, Beijing Information Science and Technology University, Beijing 100101, China 2National Key Laboratory of Complex System Simulation, Beijing 100101, China
[Objective] This article addresses the difficulties of relationship extraction due to overlapping entity relationship in military texts. [Methods] We used the BERT model as the encoder for the input texts, and used the hierarchical reinforcement learning approach to decode relationship and their corresponding entities. Then, we merged the relational position features in the entity decoding process to construct a relationship extraction model for military domains. [Results] The F1 value reached 82.2% on the military weapon and equipment dataset, which was about 8% higher than other methods. Using the publicly available NYT10 and NYT10-sub datasets, the F1 values reached 71.8% and 69.0%, which was about 7% and 9% higher than other methods. [Limitations] The new method’s extraction performance is better on manually annotated datasets. More research is needed to improve it performance on remotely supervised datasets with much noise. [Conclusions] The HBP method could effectively extract relationship among the military domains, and has some generalization potentiality.
马江微, 吕学强, 游新冬, 肖刚, 韩君妹. 融合BERT与关系位置特征的军事领域关系抽取方法*[J]. 数据分析与知识发现, 2021, 5(8): 1-12.
Ma Jiangwei, Lv Xueqiang, You Xindong, Xiao Gang, Han Junmei. Extracting Relationship Among Military Domains with BERT and Relation Position Features. Data Analysis and Knowledge Discovery, 2021, 5(8): 1-12.
( Lin Wangqun, Wang Miao, Wang Wei, et al. A Survey to Knowledge Graph and Its Military Application[J]. Journal of Chinese Information Processing, 2020, 34(12):9-16.)
( E Haihong, Zhang Wenjing, Xiao Siqi, et al. Survey of Entity Relationship Extraction Based on Deep Learning[J]. Journal of Software, 2019, 30(6):1793-1818.)
[3]
Mintz M, Bills S, Snow R, et al. Distant Supervision for Relation Extraction without Labeled Data [C]// Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP. 2009: 1003-1011.
[4]
Gormley M R, Yu M, Dredze M. Improved Relation Extraction with Feature-Rich Compositional Embedding Models [C]// Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. 2015: 1774-1784.
[5]
Zheng S C, Wang F, Bao H Y, et al. Joint Extraction of Entities and Relations Based on a Novel Tagging Scheme [C]// Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics. 2017: 1227-1236.
[6]
Takanobu R, Zhang T Y, Liu J X, et al. A Hierarchical Framework for Relation Extraction with Reinforcement Learning [C]// Proceedings of the AAAI Conference on Artificial Intelligence. 2019: 7072-7079.
[7]
Devlin J, Chang M W, Lee K, et al. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding [C]// Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2019: 4171-4186.
[8]
Lample G, Ballesteros M, Subramanian S, et al. Neural Architectures for Named Entity Recognition [C]// Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2016: 260-270.
[9]
Santos C N D, Xiang B, Zhou B W. Classifying Relations by Ranking with Convolutional Neural Networks [C]// Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing. 2015: 626-634.
( Li Weijiang, Li Tao, Qi Fang. Chinese Entity Relation Extraction Based on Multi-Features Self-Attention Bi-LSTM[J]. Journal of Chinese Information Processing, 2019, 33(10):47-56.)
( Wu Yuemin, Ding Ganggui, Hu Bin. Extracting Relationship of Agricultural Financial Texts with Attention Mechanism[J]. Data Analysis and Knowledge Discovery, 2019, 3(5):86-92.)
[14]
Li Q, Ji H. Incremental Joint Extraction of Entity Mentions and Relations [C]// Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics. 2014: 402-412.
[15]
Ren X, Wu Z Q, He W Q, et al. CoType: Joint Extraction of Typed Entities and Relations with Knowledge Bases [C]// Proceedings of the 26th International Conference on World Wide Web. 2017: 1015-1024.
[16]
Miwa M, Bansal M. End-to-End Relation Extraction Using LSTMs on Sequences and Tree Structures [C]// Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics. 2016: 1105-1116.
[17]
Bekoulis G, Deleu J, Demeester T, et al. Joint Entity Recognition and Relation Extraction as a Multi-head Selection Problem[J]. Expert Systems with Applications, 2018, 114:34-45.
doi: 10.1016/j.eswa.2018.07.032
[18]
Li X Y, Yin F, Sun Z J, et al. Entity-Relation Extraction as Multi-Turn Question Answering [C]// Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. 2019: 1340-1350.
[19]
Wei Z P, Su J L, Wang Y, et al. A Novel Cascade Binary Tagging Framework for Relational Triple Extraction [C]// Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 2020: 1476-1488.
[20]
Zeng D J, Liu K, Lai S, et al. Relation Classification via Convolutional Deep Neural Network [C]// Proceedings of the 25th International Conference on Computational Linguistics. 2014: 2335-2344.
[21]
Ji G, Liu K, He S W, et al. Distant Supervision for Relation Extraction with Sentence-level Attention and Entity Descriptions [C]// Proceedings of the 31st AAAI Conference on Artificial Intelligence. 2017: 3060-3066.
( Li Qianqian, Zhang Keliang. Entity Relation Extraction Based on Dependency Parsing in Military Field[J]. Technology Intelligence Engineering, 2019, 5(1):98-112.)
( Tian Jialai, Lü Xueqiang, You Xindong, et al. Joint Extraction of Entities and Relations Based on Hierarchical Sequence Labeling[J]. Acta Scientiarum Naturalium Universitatis Pekinensis, 2021, 57(1):53-60.)
[24]
Vaswani A, Shazeer N, Parmar N, et al. Attention is All You Need[J]. Advances in Neural Information Processing Systems, 2017, 30:5998-6008.
( Li Peiyun, Li Maoxi, Qiu Bailian, et al. Integrating BERT Word Embedding into Quality Estimation of Machine Translation[J]. Journal of Chinese Information Processing, 2020, 34(3):56-63.)
[26]
Sutton R S, Barto A G. Reinforcement Learning: An Introduction[M]. MIT Press, 1998.
[27]
Bishop C M. Pattern Recognition and Machine Learning[M]. Springer, 2006.
[28]
van Rijsbergen C J. Information Retrieval[M]. Butterworths, 1975.
[29]
Sutton R S, McAllester D A, Singh S P, et al. Policy Gradient Methods for Reinforcement Learning with Function Approximation[A]// Advances in Neural Information Processing Systems[M]. MIT Press, 2000: 1057-1063.
[30]
Riedel S, Yao L M, McCallum A. Modeling Relations and Their Mentions without Labeled Text [C]// Proceedings of Joint European Conference on Machine Learning and Knowledge Discovery in Databases. Springer, Berlin, Heidelberg, 2010: 148-163.
[31]
Zeng X R, Zeng D J, He S Z, et al. Extracting Relational Facts by an End-to-end Neural Model with Copy Mechanism [C]// Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics. 2018: 506-514.
[32]
Fu T J, Li P H, Ma W Y. GraphRel: Modeling Text as Relational Graphs for Joint Entity and Relation Extraction [C]// Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. 2019: 1409-1418.
[33]
Su J L. Hybrid Structure of Pointer and Tagging for Relation Extraction: A Baseline[EB/OL].(2019-06-03).[2021-02-24]. https://github.com/bojone/kg-2019-baseline.