[Objective] This study proposes a new classification method for social media images by fusing the related texts, aiming to efficiently construct emergency portrait. [Methods] First, we analyzed the general process of creating emergency portrait based on social media. Then, we designed a two-layer image classification system for the dimensions of emergency portrait. Third, we proposed a deep learning model (Unimodal and Crossmodal Transformer Model, UCTM) for image classification, which integrated image and text multimodal semantics. We constructed emergency portrait with our model on the dataset of Super Typhoon Mangkhut, and compared its performance with the existing ones. [Results] The MAP score of our UCTM was 0.021 higher than those of single-modal classification methods and bilinear fusion methods. For the preparation and rescue information, the F1 scores of our algorithm were 0.017 and 0.018 better than the direct classification ones. [Limitations] Our model does not investigate the inconsistency between textual and graphic semantics, and the types of emergencies need to be expanded. [Conclusions] This proposed method enriches the dimensions and contents of emergency portrait, which improves the preparation and response for crisis.
李纲, 张霁, 毛进. 面向突发事件画像的社交媒体图像分类研究*[J]. 数据分析与知识发现, 2022, 6(2/3): 67-79.
Li Gang, Zhang Ji, Mao Jin. Social Media Image Classification for Emergency Portrait. Data Analysis and Knowledge Discovery, 2022, 6(2/3): 67-79.
( Kuang Qiyu. The Study on the Effect of Social Media in Catastrophe Risk Governance[D]. Chengdu: Southwestern University of Finance and Economics, 2014.)
[2]
Gao H J, Barbier G, Goolsby R. Harnessing the Crowdsourcing Power of Social Media for Disaster Relief[J]. IEEE Intelligent Systems, 2011, 26(3):10-14.
( Chen Hongyu. Visual Dissemination and Media Power in Consumer Age[J]. Nanjing Journal of Social Sciences, 2014(11):103-108.)
[5]
Kumar A, Singh J P, Dwivedi Y K, et al. A Deep Multi-Modal Neural Network for Informative Twitter Content Classification During Emergencies[J]. Annals of Operations Research, 2020: 1-32.
[6]
Panem S, Gupta M, Varma V. Structured Information Extraction from Natural Disaster Events on Twitter[C]// Proceedings of the 5th International Workshop on Web-scale Knowledge Representation Retrieval & Reasoning. ACM Press, 2014: 1-8.
[7]
Wang Y, Hou X. A Method for Constructing Knowledge Graph of Disaster News Based on Address Tree[C]// Proceedings of the 5th International Conference on Systems and Informatics (ICSAI). IEEE, 2018: 305-310.
[8]
Imran M, Elbassuoni S, Castillo C, et al. Extracting Information Nuggets from Disaster-Related Messages in Social Media[C]// Proceedings of the 10th International Conference on Information Systems for Crisis Response and Management. IEEE, 2013.
[9]
Ni J X, Liu X, Zhou Q F, et al. A Knowledge Graph Based Disaster Storyline Generation Framework[C]// Proceedings of the 2019 Chinese Control and Decision Conference (CCDC). IEEE, 2019: 4432-4437.
[10]
Chowdhury S R, Imran M, Asghar M R, et al. Tweet4act: Using Incident-specific Profiles for Classifying Crisis-related Messages[C]// Proceedings of the 10th International Conference on Information Systems for Crisis Response and Management. IEEE, 2013.
[11]
Basu M, Shandilya A, Khosla P, et al. Extracting Resource Needs and Availabilities from Microblogs for Aiding Post-Disaster Relief Operations[J]. IEEE Transactions on Computational Social Systems, 2019, 6(3):604-618.
doi: 10.1109/TCSS.2019.2914179
[12]
马超. 多模态环境下的突发事件信息组织研究[D]. 武汉: 武汉大学, 2020.
[12]
( Ma Chao. Research on Emergency Information Organization in Multimodal Environment[D]. Wuhan: Wuhan University, 2020.)
[13]
Nguyen D T, Ofli F, Imran M, et al. Damage Assessment from Social Media Imagery Data During Disasters[C]// Proceedings of the 2017 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining 2017. ACM, 2017: 569-576.
[14]
Cirneanu A L, Popescu D, Ichim L. CNN Based on LBP for Evaluating Natural Disasters[C]// Proceedings of the 15th International Conference on Control, Automation, Robotics and Vision (ICARCV). IEEE, 2018: 568-573.
[15]
Weber E, Marzo N, Papadopoulos D P, et al. Detecting Natural Disasters, Damage, and Incidents in the Wild[C]// Proceedings of the 16th European Conference on Computer Vision. IEEE, 2020: 331-350.
[16]
Alam F, Ofli F, Imran M. CrisisMMD: Multimodal Twitter Datasets from Natural Disasters[OL]. arXiv Preprint, arXiv:1805.00713.
[17]
Ofli F, Alam F, Imran M. Analysis of Social Media Data Using Multimodal Deep Learning for Disaster Response[OL]. arXiv Preprint, arXiv:2004.11838.
( Liu Zongtian, Huang Meili, Zhou Wen, et al. Research on Event-oriented Ontology Model[J]. Computer Science, 2009, 36(11):189-192, 199.)
[19]
范维澄, 闪淳昌. 公共安全与应急管理[M]. 北京: 科学出版社, 2017.
[19]
( Fan Weicheng, Shan Chunchang. Public Safety and Emergency Management[M]. Beijing: Science Press, 2017.)
[20]
Cooper A. The Inmates are Running the Asylum: Why High-tech Products Drive Us Crazy and How to Restore the Sanity[M]. Indianapolis: SAMS, 2004.
[21]
Fohringer J, Dransch D, Kreibich H, et al. Social Media as an Information Source for Rapid Flood Inundation Mapping[J]. Natural Hazards and Earth System Sciences, 2015, 15(12):2725-2738.
[22]
Huang G, Liu Z, van der Maaten L, et al. Densely Connected Convolutional Networks[C]// Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 2017: 2261-2269.
[23]
Ren S Q, He K M, Girshick R, et al. Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39(6):1137-1149.
doi: 10.1109/TPAMI.2016.2577031
[24]
Deng J, Dong W, Socher R, et al. ImageNet: A Large-Scale Hierarchical Image Database[C]// Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 2009: 248-255.
[25]
Chen K, Wang J, Pang J, et al. MMDetection: Open MMLab Detection Toolbox and Benchmark[OL]. arXiv Preprint, arXiv:1906.07155.
[26]
Lin T Y, Maire M, Belongie S, et al. Microsoft COCO: Common Objects in Context[C]// Proceedings of the 2014 European Conference on Computer Vision. Springer, 2014: 740-755.
[27]
Vaswani A, Shazeer N, Parmar N, et al. Attention is All You Need[OL]. arXiv Preprint, arXiv:1706.03762.
[28]
Devlin J, Chang M W, Lee K, et al. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding[OL]. arXiv Preprint, arXiv:1810.04805.
[29]
Baum E B. On the Capabilities of Multilayer Perceptrons[J]. Journal of Complexity, 1988, 4(3):193-215.
doi: 10.1016/0885-064X(88)90020-9
[30]
李显鑫. 基于图文相关的突发事件图片采集研究[D]. 武汉: 武汉大学, 2020.
[30]
( Li Xianxin. Research on Picture Collection of Emergencies Based on the Correlation Between Pictures and Texts[D]. Wuhan: Wuhan University, 2020.)
[31]
Shim H. PHash: A Memory-efficient, High-performance Key-value Store for Large-scale Data-intensive Applications[J]. Journal of Systems and Software, 2017, 123:33-44.
doi: 10.1016/j.jss.2016.09.047
[32]
Zadeh A, Chen M H, Poria S, et al. Tensor Fusion Network for Multimodal Sentiment Analysis[C]// Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. 2017: 1103-1114.
( Li Gang, Zhang Ji, Mao Jin, et al. Relevance Between Image and Text of Social Media Posts for Disasters[J]. Journal of the China Society for Scientific and Technical Information, 2020, 39(11):1223-1231.)