[Objective] This paper proposes a social media rumor detection model based on the multimodal heterogeneous graph, aiming to verify the correlation between different rumor modalities and improve the accuracy of rumor detection. [Methods] First, we retrieved multimodal posts from social platforms. Then, we extracted feature representations of texts, pictures, and user attributes through preprocessing. Third, we constructed a heterogeneous graph based on the correlation between texts, pictures, and users. Fourth, we extracted the embeddings of text-type nodes according to their specified meta path. Finally, we input the embedding into the classifier to determine whether or not it is a rumor. [Results] We examined the proposed model with two open data sets. The accuracy of our model reached 91.3% and 93.8%, which were also higher than the baseline models. [Limitations] The three types of nodes from the sharing multimodal rumors will make the heterogeneous graph sparse. The proposed model is more suitable for small topic communities. [Conclusions] There is a correlation between different modalities of rumors, which helps the proposed model effectively detect multimodal rumors.
强子珊, 顾益军. 基于多模态异质图的社交媒体谣言检测模型*[J]. 数据分析与知识发现, 2023, 7(11): 68-78.
Qiang Zishan, Gu Yijun. Detecting Social Media Rumors Based on Multimodal Heterogeneous Graph. Data Analysis and Knowledge Discovery, 2023, 7(11): 68-78.
Castillo C, Mendoza M, Poblete B. Information Credibility on Twitter[C]// Proceedings of the 20th International Conference on World Wide Web. ACM, 2011: 675-684.
[2]
Wu K, Yang S, Zhu K Q. False Rumors Detection on Sina Weibo by Propagation Structures[C]// Proceedings of 2015 IEEE 31st International Conference on Data Engineering. 2015: 651-662.
[3]
Qazvinian V, Rosengren E, Radev D R, et al. Rumor Has It: Identifying Misinformation in Microblogs[C]// Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing. 2011: 1589-1599.
(Zu Kunlin, Zhao Mingwei, Guo Kai, et al. Research on the Detection of Rumor on Sina Weibo[J]. Journal of Chinese Information Processing, 2017, 31(3): 198-204.)
(Li Wenzheng, Gu Yijun, Yan Hongli. Predicting Community Numbers with Network Bayesian Information Criterion[J]. Data Analysis and Knowledge Discovery, 2020, 4(4): 72-82.)
(Wang Benyu, Gu Yijun, Peng Shufan. Semi-supervised Community Detection Algorithm Based on Particle Competition[J]. Journal of Frontiers of Computer Science and Technology, 2023, 17(3): 608-619.)
doi: 10.3778/j.issn.1673-9418.2106040
[7]
Ma J, Gao W, Mitra P, et al. Detecting Rumors from Microblogs with Recurrent Neural Networks[C]// Proceedings of the 25th International Joint Conference on Artificial Intelligence. 2016: 3818-3824.
(Liu Zheng, Wei Zhihua, Zhang Renxian. Rumor Detection Based on Convolutional Neural Network[J]. Journal of Computer Applications, 2017, 37(11): 3053-3056, 3100.)
doi: 10.11772/j.issn.1001-9081.2017.11.3053
[9]
Ma J, Gao W, Wong K F. Rumor Detection on Twitter with Tree-Structured Recursive Neural Networks[C]// Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics(Volume 1:Long Papers). 2018: 1980-1989.
[10]
Bian T, Xiao X, Xu T Y, et al. Rumor Detection on Social Media with Bi-directional Graph Convolutional Networks[C]// Proceedings of the AAAI Conference on Artificial Intelligence. 2020: 549-556.
[11]
王雷全. 基于图模型的多模态社会媒体分析[D]. 北京: 北京邮电大学, 2016.
[11]
(Wang Leiquan. Analysis of Multi-modal Social Media Based on Graph Model[D]. Beijing: Beijing University of Posts and Telecommunications, 2016.)
(Wu Youzheng, Li Haoran, Yao Ting, et al. A Survey of Multimodal Information Processing Frontiers: Application, Fusion and Pre-training[J]. Journal of Chinese Information Processing, 2022, 36(5): 1-20.)
[13]
Ngiam J, Khosla A, Kim M, et al. Multimodal Deep Learning[C]// Proceedings of the 28th International Conference on International Conference on Machine Learning. 2011: 689-696.
[14]
Lu J S, Yang J W, Batra D, et al. Hierarchical Question-Image Co-attention for Visual Question Answering[C]// Proceedings of the 30th International Conference on Neural Information Processing Systems. 2016: 289-297.
[15]
Nam H, Ha J W, Kim J. Dual Attention Networks for Multimodal Reasoning and Matching[C]// Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. 2017: 2156-2164.
[16]
Wöllmer M, Metallinou A, Eyben F, et al. Context-Sensitive Multimodal Emotion Recognition from Speech and Facial Expression Using Bidirectional LSTM Modeling[C]// Proceedings of the 11th Annual Conference of the International Speech Communication Association. 2010: 2362-2365.
[17]
Hu J W, Liu Y C, Zhao J M, et al. MMGCN: Multimodal Fusion via Deep Graph Convolution Network for Emotion Recognition in Conversation[C]// Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1:Long Papers). 2021: 5666-5675.
[18]
Jin Z W, Cao J, Guo H, et al. Multimodal Fusion with Recurrent Neural Networks for Rumor Detection on Microblogs[C]// Proceedings of the 25th ACM International Conference on Multimedia. 2017: 795-816.
[19]
刘金硕, 冯阔, Jeff Z. Pan, 等. MSRD: 多模态网络谣言检测方法[J]. 计算机研究与发展, 2020, 57(11): 2328-2336.
[19]
(Liu Jinshuo, Feng Kuo, Jeff Z. Pan, et al. MSRD: Multi-modal Web Rumor Detection Method[J]. Journal of Computer Research and Development, 2020, 57(11): 2328-2336.)
(Meng Jiana, Wang Xiaopei, Li Ting, et al. Cross-Modal Rumor Detection Based on Adversarial Neural Networks[J]. Data Analysis and Knowledge Discovery, 2022, 6(12): 32-42.)
(Zhang Shaoqin, Du Shengdong, Zhang Xiaobo, et al. Social Rumor Detection Method Based on Multimodal Fusion[J]. Computer Science, 2021, 48(5): 117-123.)
doi: 10.11896/jsjkx.200400057
(Tang Yue, Ma Jing. A Rumor Detection Method Based on Enhance Adversarial Network and Multimodal Fusion[J]. Information Science, 2022, 40(6): 108-114.)
(He Jun, Zhang Caiqing, Li Xiaozhen, et al. Survey of Research on Multimodal Fusion Technology for Deep Learning[J]. Computer Engineering, 2020, 46(5): 1-11.)
doi: 10.19678/j.issn.1000-3428.0057370
[27]
Wang X, Ji H Y, Shi C, et al. Heterogeneous Graph Attention Network[C]// Proceedings of WWW’19. 2019: 2022-2032.
[28]
Zhang C X, Song D J, Huang C, et al. Heterogeneous Graph Neural Network[C]// Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 2019: 793-803.
[29]
Fu X Y, Zhang J N, Meng Z Q, et al. MAGNN: Metapath Aggregated Graph Neural Network for Heterogeneous Graph Embedding[C]// Proceedings of the WWW 2020. 2020: 2331-2341.
[30]
李昊杰. 异质图神经网络研究及其在推荐系统中的应用[D]. 成都: 电子科技大学, 2022.
[30]
(Li Haojie. Research on Heterogeneous Graph Neural Network and Its Application in Recommendation System[D]. Chengdu: University of Electronic Science and Technology of China, 2022.)
(Li Liangxun. Research and Implementation of Heterogeneous Graph Embedding for Android Malware Detection[D]. Beijing: Beijing University of Posts and Telecommunications, 2021.)
[32]
Huang Q, Yu J S, Wu J, et al. Heterogeneous Graph Attention Networks for Early Detection of Rumors on Twitter[C]// Proceedings of 2020 International Joint Conference on Neural Networks. 2020: 1-8.
(Bi Bei, Pan Huiyao, Chen Feng, et al. Microblog Rumor Detection Model Based on Heterogeneous Graph Attention Network[J]. Journal of Computer Applications, 2021, 41(12): 3546-3550.)
doi: 10.11772/j.issn.1001-9081.2021060981
[34]
Sun Z, Deng Z H, Nie J Y, et al. RotatE: Knowledge Graph Embedding by Relational Rotation in Complex Space[OL]. arXiv Preprint, arXiv:1902.10197.
[35]
Boididou C, Andreadou K, Papadopoulos S, et al. Verifying Multimedia Use at MediaEval 2015[C]// Proceedings of MediaEval 2015 Workshop. 2015.
[36]
Maigrot C, Claveau V, Kijak E, et al. MediaEval 2016: A Multimodal System for the Verifying Multimedia Use Task[C]// Proceedings of MediaEval 2016 Workshop. 2016.
(Wang Zhuang, Sui Jie. Multimodal Rumor Detection Model Based on Multilevel Fusion[J]. Computer Engineering and Design, 2022, 43(6): 1756-1761.)
[38]
Godbole S, Sarawagi S. Discriminative Methods for Multi-labeled Classification[C]// Proceedings of Pacific-Asia Conference on Knowledge Discovery and Data Mining. 2004: 22-30.
