Please wait a minute...
Data Analysis and Knowledge Discovery  2022, Vol. 6 Issue (12): 32-42    DOI: 10.11925/infotech.2096-3467.2022.0064
Current Issue | Archive | Adv Search |
Cross-Modal Rumor Detection Based on Adversarial Neural Network
Meng Jiana,Wang Xiaopei,Li Ting,Liu Shuang(),Zhao Di
School of Computer Science and Engineering, Dalian Minzu University, Dalian 116600, China
Download: PDF (2235 KB)   HTML ( 19 ) 
Export: BibTeX | EndNote (RIS)      
Abstract  

[Objective] This paper proposes an adversarial neural network model combining the text and image data, aiming to improve the effectiveness of rumor detection. [Methods] First, we integrated the self-attention mechanism with the Bi-directional Long Short-Term Memory network (BiLSTM) model to represent the text features. Then, we used the pre-trained VGG19 network model to represent the image features. Finally, we used the adversarial neural network to study the events’ common features. [Results] It is superior to the existing baseline models in terms of accuracy, precision, recall and F1 scores. The accuracy on Weibo and Twitter data sets is 3.6% and 3.5%, higher than the best result compared with the baseline models respectively. [Limitations] More research is needed to examine the feature association between the modal information, and bridge the semantic gap of cross-modal data. [Conclusions] The proposed model could more effectively learn feature representation and detect rumors.
Key words： Rumor Detection      Adversarial Neural Network      Bi-directional Long Short-Term Memory      Self-Attentional Mechanism      VGG19     
Received: 21 January 2022      Published: 03 February 2023
ZTFLH:  TP391  
Fund:National Natural Science Foundation of China(61876031);Natural Science Foundation of Liaoning Province, China(2022-BS-104)
Corresponding Authors: Liu Shuang，ORCID： 0000-0002-0095-4328     E-mail: liushuang@dlnu.edu.cn
Service
E-mail this article
Add to my bookshelf
Add to citation manager
E-mail Alert
RSS
Articles by authors
Jiana Meng
Xiaopei Wang
Ting Li
Shuang Liu
Di Zhao

Cite this article:

Meng Jiana, Wang Xiaopei, Li Ting, Liu Shuang, Zhao Di. Cross-Modal Rumor Detection Based on Adversarial Neural Network. Data Analysis and Knowledge Discovery, 2022, 6(12): 32-42.

URL:

https://manu44.magtech.com.cn/Jwk_infotech_wk3/EN/10.11925/infotech.2096-3467.2022.0064     OR     https://manu44.magtech.com.cn/Jwk_infotech_wk3/EN/Y2022/V6/I12/32

The Model Framework
数据集 标签 数量 总计
微博 真 4 749 9 528
假 4 779
推特 真 7 021 12 995
假 5 974
Statistical Information of Data
数据集 模型 准确率 F1值 精确率 召回率
微博 TextCNN 0.764 0.748 0.827 0.683
BiLSTM-Attention 0.787 0.763 0.851 0.692
推特 TextCNN 0.532 0.568 0.598 0.541
BiLSTM-Attention 0.585 0.629 0.642 0.618
Single Text Mode of Rumour Detection Results
数据集 模型 准确率 F1值 精确率 召回率
微博 ResNet152 0.690 0.675 0.705 0.647
RCNN 0.689 0.690 0.690 0.700
VGG19 0.730 0.698 0.789 0.626
推特 ResNet152 0.591 0.558 0.731 0.452
RCNN 0.590 0.570 0.720 0.470
VGG19 0.596 0.593 0.695 0.518
Rumour Detection Results of Single Visual Mode
数据集 模型 准确率 F1值 精确率 召回率
微博 VQA 0.736 0.706 0.797 0.634
Att-RNN 0.772 0.789 0.778 0.799
MSRD 0.794 0.779 0.854 0.716
DCNN 0.803 0.799 0.801 0.809
本文 0.839 0.835 0.853 0.818
推特 VQA 0.631 0.611 0.765 0.509
Att-RNN 0.664 0.676 0.749 0.615
MSRD 0.685 0.678 0.725 0.636
DCNN - - - -
本文 0.720 0.725 0.832 0.643
Cross-Modal Rumor Detection Results
数据集 模型 准确率 F1值 精确率 召回率
微博 Ours 0.839 0.835 0.853 0.818
Ours w/o adv 0.806 0.812 0.816 0.809
Ours w/o VGG19 0.774 0.775 0.775 0.774
Ours w/o BiLSTM-Attention 0.757 0.746 0.756 0.738
推特 Ours 0.720 0.725 0.832 0.643
Ours w/o adv 0.656 0.629 0.824 0.509
Ours w/o VGG19 0.620 0.604 0.615 0.606
Ours w/o BiLSTM-Attention 0.592 0.597 0.604 0.590
Ablation Results
Training Set Loss， Test Set Loss and Adversarial Loss in Microblog Dataset
Training Set Loss， Test Set Loss and Adversarial Loss in Twitter Dataset
Visual Comparison of Text Feature Representations Learned on the Microblog Test Set
Visual Comparison of Text Feature Representations Learned on the Twitter Test Set
[1] 谭荧, 张进, 夏立新. 社交媒体情境下的情感分析研究综述[J]. 数据分析与知识发现, 2020, 4(1):1-11.
[1] (Tan Ying, Zhang Jin, Xia Lixin. A Survey of Sentiment Analysis on Social Media[J]. Data Analysis and Knowledge Discovery, 2020, 4(1): 1-11.)
[2] Castillo C, Mendoza M, Poblete B. Information Credibility on Twitter[C]// Proceedings of the 20th International Conference on World Wide Web. 2011: 675-684.
[3] Wu K, Yang S, Zhu K Q. False Rumors Detection on Sina Weibo by Propagation Structures[C]// Proceedings of the 31st International Conference on Data Engineering. 2015: 651-662.
[4] Rashkin H, Choi E, Jang J Y, et al. Truth of Varying Shades: Analyzing Language in Fake News and Political Fact-Checking[C]// Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. 2017: 2931-2937.
[5] Gupta A, Lamba H, Kumaraguru P, et al. Faking Sandy: Characterizing and Identifying Fake Images on Twitter During Hurricane Sandy[C]// Proceedings of the 22nd International Conference on World Wide Web. 2013: 729-736.
[6] Guo H, Cao J, Zhang Y Z, et al. Rumor Detection with Hierarchical Social Attention Network[C]// Proceedings of the 27th ACM International Conference on Information and Knowledge Management. 2018: 943-951.
[7] 蒋雨肖, 丁晟春, 吴鹏. 基于BiLSTM-VGG16的多模态信息特征分类研究[J]. 情报理论与实践, 2021, 44(11): 180-186, 179.
[7] (Jiang Yuxiao, Ding Shengchun, Wu Peng. A Study on the Classification of Features of Multi-Modal Information Based on BiLSTM-VGG16[J]. Information Studies: Theory & Application, 2021, 44(11): 180-186, 179.)
[8] 李莎, 张怀文, 钱胜胜, 等. 多模态多层次事件网络的谣言检测[J]. 中国图象图形学报, 2021, 26(7): 1648-1657.
[8] (Li Sha, Zhang Huaiwen, Qian Shengsheng, et al. Multi-Modal Multi-Level Event Network for Rumor Detection[J]. Journal of Image and Graphics, 2021, 26(7): 1648-1657.)
[9] Jin Z W, Cao J, Guo H, et al. Multimodal Fusion with Recurrent Neural Networks for Rumor Detection on Microblogs[C]// Proceedings of the 25th ACM International Conference on Multimedia. 2017: 795-816.
[10] 宋之杰, 王建, 贾杰. 基于 SIHR1R2的突发事件谣言传播研究[J]. 情报杂志, 2016, 35(3):118-124, 136.
[10] (Song Zhijie, Wang Jian, Jia Jie. Research on Rumor Spreading in Emergency Based on SIHR1R2 Model[J]. Journal of Intelligence, 2016, 35(3):118-124, 136.)
[11] Kwon S, Cha M, Jung K, et al. Prominent Features of Rumor Propagation in Online Social Media[C]// Proceedings of the 13th International Conference on Data Mining. 2013: 1103-1108.
[12] Tong S, Koller D. Support Vector Machine Active Learning with Applications to Text Classification[J]. Journal of Machine Learning Research. 2002, 2:45-66.
[13] Ma J, Gao W, Mitra P, et al. Detecting Rumors from Microblogs with Recurrent Neural Networks[C]// Proceedings of the 25th International Joint Conference on Artificial Intelligence. 2016: 3818-3824.
[14] 刘政, 卫志华, 张韧弦. 基于卷积神经网络的谣言检测[J]. 计算机应用, 2017, 37(11): 3053-3056.
doi: 10.11772/j.issn.1001-9081.2017.11.3053
[14] (Liu Zheng, Wei Zhihua, Zhang Renxian. Rumor Detection Based on Convolutional Neural Network[J]. Journal of Computer Applications, 2017, 37(11): 3053-3056.)
doi: 10.11772/j.issn.1001-9081.2017.11.3053
[15] Chen T, Li X, Yin H Z, et al. Call Attention to Rumors: Deep Attention Based Recurrent Neural Networks for Early Rumor Detection[C]// Proceedings of Pacific-Asia Conference on Knowledge Discovery and Data Mining. 2018: 40-52.
[16] Popat K, Mukherjee S, Yates A, et al. DeClarE: Debunking Fake News and False Claims Using Evidence-Aware Deep Learning[OL]. arXiv Preprint, arXiv: 1809.06416.
[17] Wang Y Q, Ma F L, Jin Z W, et al. EANN: Event Adversarial Neural Networks for Multi-Modal Fake News Detection[C]// Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 2018: 849-857.
[18] Yang X Y, Lyu Y F, Tian T, et al. Rumor Detection on Social Media with Graph Structured Adversarial Learning[C]// Proceedings of the 29th International Conference on Artificial Intelligence. 2021: 1417-1423.
[19] Ni S W, Li J W, Kao H Y. Rumor Detection on Social Media with Hierarchical Adversarial Training[OL]. arXiv Preprint, arXiv: 2110.00425.
[20] 李奥, 但志平, 董方敏, 等. 基于改进生成对抗网络的谣言检测方法[J]. 中文信息学报, 2020, 34(9): 78-88.
[20] (Li Ao, Dan Zhiping, Dong Fangmin, et al. An Improved Generative Adversarial Network for Rumor Detection[J]. Journal of Chinese Information Processing, 2020, 34(9): 78-88.)
[21] Ganin Y, Lempitsky V. Unsupervised Domain Adaptation by Backpropagation[C]// Proceedings of the 32nd International Conference on Machine Learning. 2015: 1180-1189.
[22] Ganin Y, Ustinova E, Ajakan H, et al. Domain-Adversarial Training of Neural Networks[J]. The Journal of Machine Learning Research, 2016, 17(1): 2096-2030.
[23] Maigrot C, Claveau V, Kijak E, et al. MediaEval 2016:A Multimodal System for the Verifying Multimedia Use Task[C]// Proceedings of the 2016 MediaEval Workshop. 2016.
[24] Chen Y H. Convolutional Neural Network for Sentence Classification[D]. Waterloo, ON: University of Waterloo, 2015.
[25] Schuster M, Paliwal K K. Bidirectional Recurrent Neural Networks[J]. IEEE Transactions on Signal Processing, 1997, 45(11): 2673-2681.
doi: 10.1109/78.650093
[26] Antol S, Agrawal A, Lu J S, et al. VQA: Visual Question Answering[C]// Proceedings of 2015 IEEE International Conference on Computer Vision. 2015: 2425-2433.
[27] 刘金硕, 冯阔, Jeff Z. Pan, 等. MSRD: 多模态网络谣言检测方法[J]. 计算机研究与发展, 2020, 57(11): 2328-2336.
[27] (Liu Jinshuo, Feng Kuo, Pan J Z, et al. MSRD: Multi-Modal Web Rumor Detection Method[J]. Journal of Computer Research and Development, 2020, 57(11): 2328-2336.)
[28] 陈志毅, 隋杰. 基于DeepFM和卷积神经网络的集成式多模态谣言检测方法[J]. 计算机科学, 2022, 49(1): 101-107.
doi: 10.11896/jsjkx.201200007
[28] (Chen Zhiyi, Sui Jie. DeepFM and Convolutional Neural Networks Ensembles for Multimodal Rumor Detection[J]. Computer Science, 2022, 49(1): 101-107.)
doi: 10.11896/jsjkx.201200007
[1] Yang Hanxun, Zhou Dequn, Ma Jing, Luo Yongcong. Detecting Rumors with Uncertain Loss and Task-level Attention Mechanism[J]. 数据分析与知识发现, 2021, 5(7): 101-110.
[2] Kan Liu,Haochen Du. Detecting Twitter Rumors with Deep Transfer Network[J]. 数据分析与知识发现, 2019, 3(10): 47-55.
[3] Shou Huanrong,Deng Shuqing,Xu Jian. Detecting Online Rumors with Sentiment Analysis[J]. 数据分析与知识发现, 2017, 1(7): 44-51.
  Copyright © 2016 Data Analysis and Knowledge Discovery   Tel/Fax:(010)82626611-6626，82624938   E-mail:jishu@mail.las.ac.cn