Please wait a minute...
Data Analysis and Knowledge Discovery  2022, Vol. 6 Issue (8): 41-51    DOI: 10.11925/infotech.2096-3467.2021.1079
Current Issue | Archive | Adv Search |
Identifying Relationship of Chinese Characters with Attention Mechanism and Convolutional Neural Network
Zhao Pengwu1,Li Zhiyi2(),Lin Xiaoqi2
1Institute of Scientific and Technical Information of China, Beijing 100038, China
2School of Economics & Management, South China Normal University, Guangzhou 510006, China
Download: PDF (1321 KB)   HTML ( 32 ) 
Export: BibTeX | EndNote (RIS)      
Abstract  

[Objective] The paper tries to identify the features and relationship of dynamic semantic information from the Chinese character entities. [Methods] First, we used the attention mechanism and improved convolution neural network model to automatically extract features from the training data of public corpus with character entity relationship. Then, we compared our model’s performance with the existing ones from the perspectives of entity relationship recognition efficiency, as well as entity relationship extraction effects and efficiency. [Results] The performance of CNN+Attention model is better than those of the SVM, LR, LSTM, BiLSTM and CNN model in prediction accuracy. Our new model is 0.92% higher in accuracy, 0.80% higher in recall and 0.86% higher in F1 value than the BiLSTM model with relatively better extraction effect. [Limitations] We need to examine our model with more sample data sets. [Conclusions] The proposed model could effectively improve the accuracy and recall of entity relationship extraction for Chinese characters.
Key words： Convolutional Neural Network      Attention Mechanism      Chinese Character      Relationship Extraction      Relationship Recognition     
Received: 22 September 2021      Published: 23 September 2022
ZTFLH:  N99  
Fund:National Social Science Fund of China(17BTQ062)
Corresponding Authors: Li Zhiyi，ORCID： 0000-0001-6407-2554     E-mail: leeds@scnu.edu.cn
Service
E-mail this article
Add to my bookshelf
Add to citation manager
E-mail Alert
RSS
Articles by authors
Pengwu Zhao
Zhiyi Li
Xiaoqi Lin

Cite this article:

Zhao Pengwu, Li Zhiyi, Lin Xiaoqi. Identifying Relationship of Chinese Characters with Attention Mechanism and Convolutional Neural Network. Data Analysis and Knowledge Discovery, 2022, 6(8): 41-51.

URL:

https://manu44.magtech.com.cn/Jwk_infotech_wk3/EN/10.11925/infotech.2096-3467.2021.1079     OR     https://manu44.magtech.com.cn/Jwk_infotech_wk3/EN/Y2022/V6/I8/41

CNN+Attention Model Structure
关系 ID
Unknown 0
父母 1
合作 2
师生 3
投资 4
对手 5
情侣 6
雇佣 7
朋友 8
亲戚 9
同学 10
上下级 11
The Relationship and Corresponding ID
Flow Chart of Research Scheme
属性 属性值
Keras版本 2.2.4
TensorFlow版本 1.10
GPU NVIDIA GeForce GTX 1080Ti
CPU AMD 2950X@3.50GHz
内存 32GB
操作环境 Linux,Ubuntu16.05
Experimental Environment
参数名称 参数值
Embedding维度 200
Epochs 250
Batch size 256
Dropout rate 0.3
Learning rate 0.001
Parameter Settings of Entity Relationship Extraction Experiment
预测 实际有关系 实际无关系
预测有关系 True Positive（TP） False Positive（FP）
预测无关系 False Negative（FN） True Negative（TN）
P-R Confusion Matrix
模型 P R F1
SVM 0.901 8 0.874 3 0.887 8
LR 0.783 3 0.764 2 0.773 6
LSTM 0.938 4 0.935 3 0.936 7
BiLSTM 0.940 8 0.932 1 0.936 3
CNN 0.926 4 0.926 4 0.864 1
CNN+Attention 0.950 0 0.940 1 0.944 9
Experimental Results of Machine Learning Models
关系 P R F1
父母 0.941 4 0.946 3 0.944 1
合作 0.957 6 0.960 2 0.959 0
师生 0.943 8 0.923 5 0.933 6
投资 0.953 9 0.933 6 0.943 6
对手 0.952 8 0.921 4 0.936 8
情侣 0.942 0 0.936 3 0.939 1
雇佣 0.965 0 0.915 3 0.939 5
朋友 0.949 2 0.880 9 0.913 8
亲戚 0.911 2 0.868 4 0.889 3
同学 0.973 3 0.921 9 0.929 7
上下级 0.971 1 0.950 5 0.960 7
CNN+Attention Entity Relationship Extraction Effect
Accuracy and Loss Rate Changes of CNN+Attention Model
向量 P R F1
CNN+Attention（随机初始化向量） 0.876 1 0.848 3 0.861 9
CNN+Attention（经训练的向量） 0.917 6 0.887 5 0.902 1
Randomization Vector and Rraining Vector Relationship Extraction Effect
PR Diagram of Entity Relationship Extraction
Average Accuracy of CNN+Attention Model
[1] Bekoulis G, Deleu J, Demeester T, et al. Joint Entity Recognition and Relation Extraction as A Multi-Head Selection Problem[J]. Expert Systems with Applications, 2018, 114: 34-45.
doi: 10.1016/j.eswa.2018.07.032
[2] Kim E K, Choi K S. Improving Distantly Supervised Relation Extraction by Knowledge Base-Driven Zero Subject Resolution[J]. IEICE Transactions on Information and Systems, 2018, E101.D(10): 2551-2558.
doi: 10.1587/transinf.2017EDL8270
[3] 贾真, 冶忠林, 尹红风, 等. 基于Tri-training与噪声过滤的弱监督关系抽取[J]. 中文信息学报, 2016, 30(4): 142-149, 158.
[3] (Jia Zhen, Ye Zhonglin, Yin Hongfeng, et al. Weakly Supervised Relation Extraction Based on Tri-training and Noise Filtering[J]. Journal of Chinese Information Processing, 2016, 30(4): 142-149, 158.)
[4] 甘丽新, 万常选, 刘德喜, 等. 基于句法语义特征的中文实体关系抽取[J]. 计算机研究与发展, 2016, 53(2): 284-302.
[4] (Gan Lixin, Wan Changxuan, Liu Dexi, et al. Chinese Named Entity Relation Extraction Based on Syntactic and Semantic Features[J]. Journal of Computer Research and Development, 2016, 53(2): 284-302.)
[5] 吴胜, 刘茂福, 胡慧君, 等. 中文文本中实体数值型关系无监督抽取方法[J]. 武汉大学学报(理学版), 2016, 62(6): 552-560.
[5] (Wu Sheng, Liu Maofu, Hu Huijun, et al. Unsupervised Extraction of Attribute-Value Entity Relation from Chinese Texts[J]. Journal of Wuhan University (Natural Science Edition), 2016, 62(6): 552-560.)
[6] 肜博辉, 付琨, 黄宇, 等. 基于多通道卷积神经网的实体关系抽取[J]. 计算机应用研究, 2017, 34(3): 689-692.
[6] (Rong Bohui, Fu Kun, Huang Yu, et al. Relation Extraction Based on Multi-Channel Convolutional Neural Network[J]. Application Research of Computers, 2017, 34(3): 689-692.)
[7] 张晓斌, 陈福才, 黄瑞阳. 基于CNN和双向LSTM融合的实体关系抽取[J]. 网络与信息安全学报, 2018, 4(9): 44-51.
[7] (Zhang Xiaobin, Chen Fucai, Huang Ruiyang. Relation Extraction Based on CNN and Bi-LSTM[J]. Chinese Journal of Network and Information Security, 2018, 4(9): 44-51.)
[8] 延浩然, 靳小龙, 贾岩涛, 等. 一种改进的实体关系抽取算法——OptMultiR[J]. 中文信息学报, 2018, 32(9): 66-74.
[8] (Yan Haoran, Jin Xiaolong, Jia Yantao, et al. An Improved Entity Relation Extraction Algorithm——OptMultiR[J]. Journal of Chinese Information Processing, 2018, 32(9): 66-74.)
[9] 车金立, 唐力伟, 邓士杰, 等. 基于双重注意力机制的远程监督中文关系抽取[J]. 计算机工程与应用, 2019, 55(20): 107-113.
doi: 10.3778/j.issn.1002-8331.1806-0438
[9] (Che Jinli, Tang Liwei, Deng Shijie, et al. Distant Supervision Chinese Relation Extraction Based on Dual Attention Mechanism[J]. Computer Engineering and Applications, 2019, 55(20): 107-113.)
doi: 10.3778/j.issn.1002-8331.1806-0438
[10] Cui Z Y, Pan L, Liu S J. Hybrid BiLSTM-Siamese Network for Relation Extraction[C]// Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems. 2019: 1907-1909.
[11] 闫跃, 霍其润, 李天昊, 等. 融合多重注意力机制的卷积神经网络文本分类设计与实现[J]. 小型微型计算机系统, 2021, 42(2): 362-367.
[11] (Yan Yue, Huo Qirun, Li Tianhao, et al. Design and Implementation of Text Classification Based on Convolutional Neural Network with Multiple Attention Mechanisms[J]. Journal of Chinese Computer Systems, 2021, 42(2): 362-367.)
[12] 张婷婷, 李卫疆, 李涛. 基于自注意力卷积神经网络的实体关系抽取[J]. 信息技术, 2022, 46(1): 11-15.
[12] (Zhang Tingting, Li Weijiang, Li Tao. Entity Relation Extraction Based on Self-Attention Convolutional Neural Network[J]. Information Technology, 2022, 46(1): 11-15.)
[13] 冯建周, 宋沙沙, 王元卓, 等. 基于改进注意力机制的实体关系抽取方法[J]. 电子学报, 2019, 47(8): 1692-1700.
doi: 10.3969/j.issn.0372-2112.2019.08.012
[13] (Feng Jianzhou, Song Shasha, Wang Yuanzhuo, et al. Entity Relation Extraction Based on Improved Attention Mechanism[J]. Acta Electronica Sinica, 2019, 47(8): 1692-1700.)
doi: 10.3969/j.issn.0372-2112.2019.08.012
[14] 张兰霞, 胡文心. 基于双向GRU神经网络和双层注意力机制的中文文本中人物关系抽取研究[J]. 计算机应用与软件, 2018, 35(11): 130-135, 189.
[14] (Zhang Lanxia, Hu Wenxin. Character Relation Extraction in Chinese Text Based on Bidirectional GRU Neural Network and Dual-attention Mechanism[J]. Computer Applications and Software, 2018, 35(11): 130-135, 189.)
[15] 黄杨琛, 贾焰, 甘亮, 等. 基于远程监督的多因子人物关系抽取模型[J]. 通信学报, 2018, 39(7): 103-112.
[15] (Huang Yangchen, Jia Yan, Gan Liang, et al. Multi-Factor Person Entity Relation Extraction Model Based on Distant Supervision[J]. Journal on Communications, 2018, 39(7): 103-112.)
[16] Lv C Y, Pan D, Li Y X, et al. A Novel Chinese Entity Relationship Extraction Method Based on the Bidirectional Maximum Entropy Markov Model[J]. Complexity, 2021, 2021: 6610965.
[17] Simonyan K, Zisserman A. Very Deep Convolutional Networks for Large-Scale Image Recognition[OL]. arXiv Preprint, arXiv: 1409.1556.
[18] Mikolov T, Chen K, Corrado G, et al. Efficient Estimation of Word Representations in Vector Space[OL]. arXiv Preprint, arXiv: 1301.3781
[19] Lecun Y, Bottou L, Bengio Y, et al. Gradient-Based Learning Applied to Document Recognition[J]. Proceedings of the IEEE, 1998, 86(11): 2278-2324.
doi: 10.1109/5.726791
[20] Mnih V, Heess N, Graves A. Recurrent Models of Visual Attention[C]// Proceedings of the 27th Annual Conference on Neural Information Processing Systems. 2014: 2204-2212.
[21] Chen X, Hsieh C J, Gong B. When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentations[OL]. arXiv Preprint, arXiv: 2106.01548.
[22] Elsayed G F, Kornblith S, Le Q V. Saccader: Improving Accuracy of Hard Attention Models for Vision[OL]. arXiv Preprint, arXiv: 1908.07644.
[23] Luong T, Pham H, Manning C D. Effective Approaches to Attention-based Neural Machine Translation[C]// Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. 2015: 2412-2421.
[24] Chinchor N, Sundheim B. MUC-5 Evaluation Metrics[C]// Proceedings of the 5th Conference on Message Understanding. 1993: 69-78.
[1] Ye Han,Sun Haichun,Li Xin,Jiao Kainan. Classification Model for Long Texts with Attention Mechanism and Sentence Vector Compression[J]. 数据分析与知识发现, 2022, 6(6): 84-94.
[2] Zhang Ruoqi, Shen Jianfang, Chen Pinghua. Session Sequence Recommendation with GNN, Bi-GRU and Attention Mechanism[J]. 数据分析与知识发现, 2022, 6(6): 46-54.
[3] Guo Fanrong, Huang Xiaoxi, Wang Rongbo, Chen Zhiqun, Hu Chuang, Xie Yimin, Si Boyu. Identifying Metaphor with Transformer and Graph Convolutional Network[J]. 数据分析与知识发现, 2022, 6(4): 120-129.
[4] Wang Yongsheng, Wang Hao, Yu Wei, Zhou Zeyu. Extracting Relationship Among Characters from Local Chronicles with Text Structures and Contents[J]. 数据分析与知识发现, 2022, 6(2/3): 318-328.
[5] Guo Hangcheng, He Yanqing, Lan Tian, Wu Zhenfeng, Dong Cheng. Identifying Moves from Scientific Abstracts Based on Paragraph-BERT-CRF[J]. 数据分析与知识发现, 2022, 6(2/3): 298-307.
[6] Xu Yuemei, Fan Zuwei, Cao Han. A Multi-Task Text Classification Model Based on Label Embedding of Attention Mechanism[J]. 数据分析与知识发现, 2022, 6(2/3): 105-116.
[7] Yang Hanxun, Zhou Dequn, Ma Jing, Luo Yongcong. Detecting Rumors with Uncertain Loss and Task-level Attention Mechanism[J]. 数据分析与知识发现, 2021, 5(7): 101-110.
[8] Yin Pengbo,Pan Weimin,Zhang Haijun,Chen Degang. Identifying Clickbait with BERT-BiGA Model[J]. 数据分析与知识发现, 2021, 5(6): 126-134.
[9] Xie Hao,Mao Jin,Li Gang. Sentiment Classification of Image-Text Information with Multi-Layer Semantic Fusion[J]. 数据分析与知识发现, 2021, 5(6): 103-114.
[10] Han Pu,Zhang Zhanpeng,Zhang Mingtao,Gu Liang. Normalizing Chinese Disease Names with Multi-feature Fusion[J]. 数据分析与知识发现, 2021, 5(5): 83-94.
[11] Duan Jianyong,Wei Xiaopeng,Wang Hao. A Multi-Perspective Co-Matching Model for Machine Reading Comprehension[J]. 数据分析与知识发现, 2021, 5(4): 134-141.
[12] Wang Yuzhu,Xie Jun,Chen Bo,Xu Xinying. Multi-modal Sentiment Analysis Based on Cross-modal Context-aware Attention[J]. 数据分析与知识发现, 2021, 5(4): 49-59.
[13] Yu Bengong, Zhang Shuwen. Aspect-Level Sentiment Analysis Based on BAGCNN[J]. 数据分析与知识发现, 2021, 5(12): 37-47.
[14] Zhou Wenyuan, Wang Mingyang, Jing Yu. Automatic Classification of Citation Sentiment and Purposes with AttentionSBGMC Model[J]. 数据分析与知识发现, 2021, 5(12): 48-59.
[15] Dai Zhihong, Hao Xiaoling. Extracting Hypernym-Hyponym Relationship for Financial Market Applications[J]. 数据分析与知识发现, 2021, 5(10): 60-70.
  Copyright © 2016 Data Analysis and Knowledge Discovery   Tel/Fax:(010)82626611-6626，82624938   E-mail:jishu@mail.las.ac.cn