|
|
|
| Social Media Image Classification for Emergency Portrait |
Li Gang,Zhang Ji,Mao Jin( ) |
| Center for Studies of Information Resources, Wuhan University, Wuhan 430072, China |
|
|
|
|
Abstract [Objective] This study proposes a new classification method for social media images by fusing the related texts, aiming to efficiently construct emergency portrait. [Methods] First, we analyzed the general process of creating emergency portrait based on social media. Then, we designed a two-layer image classification system for the dimensions of emergency portrait. Third, we proposed a deep learning model (Unimodal and Crossmodal Transformer Model, UCTM) for image classification, which integrated image and text multimodal semantics. We constructed emergency portrait with our model on the dataset of Super Typhoon Mangkhut, and compared its performance with the existing ones. [Results] The MAP score of our UCTM was 0.021 higher than those of single-modal classification methods and bilinear fusion methods. For the preparation and rescue information, the F1 scores of our algorithm were 0.017 and 0.018 better than the direct classification ones. [Limitations] Our model does not investigate the inconsistency between textual and graphic semantics, and the types of emergencies need to be expanded. [Conclusions] This proposed method enriches the dimensions and contents of emergency portrait, which improves the preparation and response for crisis.
|
|
Received: 31 August 2021
Published: 14 April 2022
|
|
|
| Fund:National Natural Science Foundation of China(71790612);National Natural Science Foundation of China(71921002) |
|
Corresponding Authors:
Mao Jin,ORCID：0000-0001-9572-6709
E-mail: danveno@163.com
|
| [1] |
邝启宇. 社交媒体在巨灾风险治理中的作用研究[D]. 成都: 西南财经大学, 2014.
|
| [1] |
( Kuang Qiyu. The Study on the Effect of Social Media in Catastrophe Risk Governance[D]. Chengdu: Southwestern University of Finance and Economics, 2014.)
|
| [2] |
Gao H J, Barbier G, Goolsby R. Harnessing the Crowdsourcing Power of Social Media for Disaster Relief[J]. IEEE Intelligent Systems, 2011, 26(3):10-14.
|
| [3] |
周锐, 黄静, 范嘉祺. 突发公共卫生事件大数据画像构建研究[J]. 电子政务, 2020(6):12-20.
|
| [3] |
( Zhou Rui, Huang Jing, Fan Jiaqi. Research on the Construction of Big Data Portrait of Public Health Emergencies[J]. E-Government, 2020(6):12-20.)
|
| [4] |
陈红玉. 消费时代的图像传播及其媒介权力[J]. 南京社会科学, 2014(11):103-108.
|
| [4] |
( Chen Hongyu. Visual Dissemination and Media Power in Consumer Age[J]. Nanjing Journal of Social Sciences, 2014(11):103-108.)
|
| [5] |
Kumar A, Singh J P, Dwivedi Y K, et al. A Deep Multi-Modal Neural Network for Informative Twitter Content Classification During Emergencies[J]. Annals of Operations Research, 2020: 1-32.
|
| [6] |
Panem S, Gupta M, Varma V. Structured Information Extraction from Natural Disaster Events on Twitter[C]// Proceedings of the 5th International Workshop on Web-scale Knowledge Representation Retrieval & Reasoning. ACM Press, 2014: 1-8.
|
| [7] |
Wang Y, Hou X. A Method for Constructing Knowledge Graph of Disaster News Based on Address Tree[C]// Proceedings of the 5th International Conference on Systems and Informatics (ICSAI). IEEE, 2018: 305-310.
|
| [8] |
Imran M, Elbassuoni S, Castillo C, et al. Extracting Information Nuggets from Disaster-Related Messages in Social Media[C]// Proceedings of the 10th International Conference on Information Systems for Crisis Response and Management. IEEE, 2013.
|
| [9] |
Ni J X, Liu X, Zhou Q F, et al. A Knowledge Graph Based Disaster Storyline Generation Framework[C]// Proceedings of the 2019 Chinese Control and Decision Conference (CCDC). IEEE, 2019: 4432-4437.
|
| [10] |
Chowdhury S R, Imran M, Asghar M R, et al. Tweet4act: Using Incident-specific Profiles for Classifying Crisis-related Messages[C]// Proceedings of the 10th International Conference on Information Systems for Crisis Response and Management. IEEE, 2013.
|
| [11] |
Basu M, Shandilya A, Khosla P, et al. Extracting Resource Needs and Availabilities from Microblogs for Aiding Post-Disaster Relief Operations[J]. IEEE Transactions on Computational Social Systems, 2019, 6(3):604-618.
doi: 10.1109/TCSS.2019.2914179
|
| [12] |
马超. 多模态环境下的突发事件信息组织研究[D]. 武汉: 武汉大学, 2020.
|
| [12] |
( Ma Chao. Research on Emergency Information Organization in Multimodal Environment[D]. Wuhan: Wuhan University, 2020.)
|
| [13] |
Nguyen D T, Ofli F, Imran M, et al. Damage Assessment from Social Media Imagery Data During Disasters[C]// Proceedings of the 2017 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining 2017. ACM, 2017: 569-576.
|
| [14] |
Cirneanu A L, Popescu D, Ichim L. CNN Based on LBP for Evaluating Natural Disasters[C]// Proceedings of the 15th International Conference on Control, Automation, Robotics and Vision (ICARCV). IEEE, 2018: 568-573.
|
| [15] |
Weber E, Marzo N, Papadopoulos D P, et al. Detecting Natural Disasters, Damage, and Incidents in the Wild[C]// Proceedings of the 16th European Conference on Computer Vision. IEEE, 2020: 331-350.
|
| [16] |
Alam F, Ofli F, Imran M. CrisisMMD: Multimodal Twitter Datasets from Natural Disasters[OL]. arXiv Preprint, arXiv:1805.00713.
|
| [17] |
Ofli F, Alam F, Imran M. Analysis of Social Media Data Using Multimodal Deep Learning for Disaster Response[OL]. arXiv Preprint, arXiv:2004.11838.
|
| [18] |
刘宗田, 黄美丽, 周文, 等. 面向事件的本体研究[J]. 计算机科学, 2009, 36(11):189-192,199.
|
| [18] |
( Liu Zongtian, Huang Meili, Zhou Wen, et al. Research on Event-oriented Ontology Model[J]. Computer Science, 2009, 36(11):189-192, 199.)
|
| [19] |
范维澄, 闪淳昌. 公共安全与应急管理[M]. 北京: 科学出版社, 2017.
|
| [19] |
( Fan Weicheng, Shan Chunchang. Public Safety and Emergency Management[M]. Beijing: Science Press, 2017.)
|
| [20] |
Cooper A. The Inmates are Running the Asylum: Why High-tech Products Drive Us Crazy and How to Restore the Sanity[M]. Indianapolis: SAMS, 2004.
|
| [21] |
Fohringer J, Dransch D, Kreibich H, et al. Social Media as an Information Source for Rapid Flood Inundation Mapping[J]. Natural Hazards and Earth System Sciences, 2015, 15(12):2725-2738.
|
| [22] |
Huang G, Liu Z, van der Maaten L, et al. Densely Connected Convolutional Networks[C]// Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 2017: 2261-2269.
|
| [23] |
Ren S Q, He K M, Girshick R, et al. Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39(6):1137-1149.
doi: 10.1109/TPAMI.2016.2577031
|
| [24] |
Deng J, Dong W, Socher R, et al. ImageNet: A Large-Scale Hierarchical Image Database[C]// Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 2009: 248-255.
|
| [25] |
Chen K, Wang J, Pang J, et al. MMDetection: Open MMLab Detection Toolbox and Benchmark[OL]. arXiv Preprint, arXiv:1906.07155.
|
| [26] |
Lin T Y, Maire M, Belongie S, et al. Microsoft COCO: Common Objects in Context[C]// Proceedings of the 2014 European Conference on Computer Vision. Springer, 2014: 740-755.
|
| [27] |
Vaswani A, Shazeer N, Parmar N, et al. Attention is All You Need[OL]. arXiv Preprint, arXiv:1706.03762.
|
| [28] |
Devlin J, Chang M W, Lee K, et al. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding[OL]. arXiv Preprint, arXiv:1810.04805.
|
| [29] |
Baum E B. On the Capabilities of Multilayer Perceptrons[J]. Journal of Complexity, 1988, 4(3):193-215.
doi: 10.1016/0885-064X(88)90020-9
|
| [30] |
李显鑫. 基于图文相关的突发事件图片采集研究[D]. 武汉: 武汉大学, 2020.
|
| [30] |
( Li Xianxin. Research on Picture Collection of Emergencies Based on the Correlation Between Pictures and Texts[D]. Wuhan: Wuhan University, 2020.)
|
| [31] |
Shim H. PHash: A Memory-efficient, High-performance Key-value Store for Large-scale Data-intensive Applications[J]. Journal of Systems and Software, 2017, 123:33-44.
doi: 10.1016/j.jss.2016.09.047
|
| [32] |
Zadeh A, Chen M H, Poria S, et al. Tensor Fusion Network for Multimodal Sentiment Analysis[C]// Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. 2017: 1103-1114.
|
| [33] |
李纲, 张霁, 毛进, 等. 灾害事件下社交媒体图文相关性研究[J]. 情报学报, 2020, 39(11):1223-1231.
|
| [33] |
( Li Gang, Zhang Ji, Mao Jin, et al. Relevance Between Image and Text of Social Media Posts for Disasters[J]. Journal of the China Society for Scientific and Technical Information, 2020, 39(11):1223-1231.)
|
|
Viewed |
|
|
|
Full text
|
|
|
|
|
Abstract
|
|
|
|
|
Cited |
|
|
|
|
| |
Shared |
|
|
|
|
| |
Discussed |
|
|
|
|
