Please wait a minute...
Advanced Search
数据分析与知识发现  2022, Vol. 6 Issue (2/3): 67-79     https://doi.org/10.11925/infotech.2096-3467.2021.0952
  专辑 本期目录 | 过刊浏览 | 高级检索 |
面向突发事件画像的社交媒体图像分类研究*
李纲,张霁,毛进()
武汉大学信息资源研究中心 武汉 430072
Social Media Image Classification for Emergency Portrait
Li Gang,Zhang Ji,Mao Jin()
Center for Studies of Information Resources, Wuhan University, Wuhan 430072, China
全文: PDF (4018 KB)   HTML ( 23
输出: BibTeX | EndNote (RIS)      
摘要 

【目的】 为有效支撑突发事件画像,融合文本信息对社交媒体图像进行分类。【方法】 分析基于社交媒体的突发事件画像基本流程,并面向突发事件画像描述维度需求设计了具有双层结构的图像分类体系,进而提出一种融合图像和文本多模态语义的图像分类深度学习模型UCTM(UniModal and CrossModal Transformer Model),在“台风山竹”事件数据集上开展对比实验,并构建了示例画像。【结果】 融合多模态信息的UCTM模型MAP指标比单模态分类方法和双线性融合方法高0.021以上;在应对信息(准备)和应对信息(救援)两个类别上,两阶段层次化分类策略的F1值比直接分类策略分别高0.017和0.018。【局限】 模型未考虑图文语义不一致的情况,实验涉及的突发事件类型较为单一。【结论】 本文方法能够丰富突发事件画像维度和内容,有助于提升突发事件态势感知的精确性和全面性。

服务
把本文推荐给朋友
加入引用管理器
E-mail Alert
RSS
作者相关文章
李纲
张霁
毛进
关键词 社交媒体突发事件图像分类多模态事件画像    
Abstract

[Objective] This study proposes a new classification method for social media images by fusing the related texts, aiming to efficiently construct emergency portrait. [Methods] First, we analyzed the general process of creating emergency portrait based on social media. Then, we designed a two-layer image classification system for the dimensions of emergency portrait. Third, we proposed a deep learning model (Unimodal and Crossmodal Transformer Model, UCTM) for image classification, which integrated image and text multimodal semantics. We constructed emergency portrait with our model on the dataset of Super Typhoon Mangkhut, and compared its performance with the existing ones. [Results] The MAP score of our UCTM was 0.021 higher than those of single-modal classification methods and bilinear fusion methods. For the preparation and rescue information, the F1 scores of our algorithm were 0.017 and 0.018 better than the direct classification ones. [Limitations] Our model does not investigate the inconsistency between textual and graphic semantics, and the types of emergencies need to be expanded. [Conclusions] This proposed method enriches the dimensions and contents of emergency portrait, which improves the preparation and response for crisis.

Key wordsSocial Media    Emergency    Image Classification    Multimodal    Event Portrait
收稿日期: 2021-08-31      出版日期: 2022-04-14
ZTFLH:  TP393  
基金资助:*国家自然科学基金重大项目(71790612);国家自然科学基金创新研究群体项目的研究成果之一(71921002)
通讯作者: 毛进,ORCID:0000-0001-9572-6709     E-mail: danveno@163.com
引用本文:   
李纲, 张霁, 毛进. 面向突发事件画像的社交媒体图像分类研究*[J]. 数据分析与知识发现, 2022, 6(2/3): 67-79.
Li Gang, Zhang Ji, Mao Jin. Social Media Image Classification for Emergency Portrait. Data Analysis and Knowledge Discovery, 2022, 6(2/3): 67-79.
链接本文:  
https://manu44.magtech.com.cn/Jwk_infotech_wk3/CN/10.11925/infotech.2096-3467.2021.0952      或      https://manu44.magtech.com.cn/Jwk_infotech_wk3/CN/Y2022/V6/I2/3/67
Fig.1  突发事件画像流程
大类 子类 说明
受灾信息 受灾载体:人 突发事件对人类生命安全和日常生活的影响,如人受伤、死亡、流离失所等
受灾载体:环境 突发事件对社会环境如基础设施和自然环境造成影响,如车辆抛锚、房屋倒塌、大坝溃堤等
应对信息 准备和预防 人们为了应对灾害而提前做的处理,如房屋加固、道路巡查等;图像对象可以包含应对行为的实施者或图像场景是因为应对行为而产生的场景(如加固后的门窗或堤坝)
救援和处理 灾害发生以及之后人们采取的应急行动,如救援、捐赠等;图像对象可以包含应对行为的实施者或图像场景是因为应对行为而产生的场景(如救援的物资、修复后的房屋)
事件基本信息 描述事件的基本属性的图像,如发生地点、时间、强度等
其他信息 不属于以上类别的图像归于此类
Table 1  突发事件图像分类体系
Fig.2  UCTM模型结构
Fig.3  UniModal和CrossModal的融合结构
Fig.4  标注数据图像类别分布
Fig.5  不同模型多次运行MAP指标
模型 指标 受灾信息(载体:人) 受灾
信息
(载体:环境)
应对
信息
(准备和预防)
应对
信息
(救援和处理)
基本
信息
DenseNet 准确率 0.769 0.848 0.671 0.694 0.899
召回率 0.128 0.864 0.314 0.539 0.934
F1值 0.216 0.855 0.426 0.605 0.916
UTM 准确率 0.677 0.864 0.675 0.701 0.937 8
召回率 0.260 0.866 0.408 0.570 0.917 8
F1值 0.374 0.865 0.508 0.629 0.927 6
Concat 准确率 0.678 0.872 0.777 0.799 0.949
召回率 0.287 0.879 0.536 0.719 0.906
F1值 0.403 0.876 0.628 0.756 0.927
TFN 准确率 0.718 0.833 0.708 0.811 0.905
召回率 0.337 0.826 0.611 0.727 0.915
F1值 0.459 0.877 0.656 0.767 0.910
UCTM 准确率 0.656 0.875 0.754 0.800 0.950
召回率 0.511 0.880 0.626 0.760 0.883
F1值 0.555 0.878 0.683 0.779 0.926
Table 2  不同模型在各个分类类别上的性能表现
Fig.6  不同模型两阶段分类效果对比
模型 分类形式 受灾信息(载体:人) 受灾
信息
(载体:环境)
应对
信息
(准备和预防)
应对
信息
(救援和处理)
基本信息
DenseNet 直接分类 0.216 0.855 0.426 0.605 0.916
层次分类 0.419 0.855 0.500 0.683 0.922
UTM 直接分类 0.414 0.866 0.498 0.631 0.934
层次分类 0.493 0.861 0.549 0.693 0.937
Concat 直接分类 0.403 0.876 0.628 0.756 0.927
层次分类 0.507 0.869 0.653 0.776 0.924
TFN 直接分类 0.459 0.877 0.656 0.767 0.910
层次分类 0.519 0.878 0.658 0.765 0.907
UCTM 直接分类 0.554 0.879 0.684 0.778 0.926
层次分类 0.547 0.871 0.701 0.796 0.901
Table 3  直接分类和二阶段分类F1比较
Fig.7  各类别错误分类图片
Fig.8  广东台风山竹灾害事件画像
[1] 邝启宇. 社交媒体在巨灾风险治理中的作用研究[D]. 成都: 西南财经大学, 2014.
[1] ( Kuang Qiyu. The Study on the Effect of Social Media in Catastrophe Risk Governance[D]. Chengdu: Southwestern University of Finance and Economics, 2014.)
[2] Gao H J, Barbier G, Goolsby R. Harnessing the Crowdsourcing Power of Social Media for Disaster Relief[J]. IEEE Intelligent Systems, 2011, 26(3):10-14.
[3] 周锐, 黄静, 范嘉祺. 突发公共卫生事件大数据画像构建研究[J]. 电子政务, 2020(6):12-20.
[3] ( Zhou Rui, Huang Jing, Fan Jiaqi. Research on the Construction of Big Data Portrait of Public Health Emergencies[J]. E-Government, 2020(6):12-20.)
[4] 陈红玉. 消费时代的图像传播及其媒介权力[J]. 南京社会科学, 2014(11):103-108.
[4] ( Chen Hongyu. Visual Dissemination and Media Power in Consumer Age[J]. Nanjing Journal of Social Sciences, 2014(11):103-108.)
[5] Kumar A, Singh J P, Dwivedi Y K, et al. A Deep Multi-Modal Neural Network for Informative Twitter Content Classification During Emergencies[J]. Annals of Operations Research, 2020: 1-32.
[6] Panem S, Gupta M, Varma V. Structured Information Extraction from Natural Disaster Events on Twitter[C]// Proceedings of the 5th International Workshop on Web-scale Knowledge Representation Retrieval & Reasoning. ACM Press, 2014: 1-8.
[7] Wang Y, Hou X. A Method for Constructing Knowledge Graph of Disaster News Based on Address Tree[C]// Proceedings of the 5th International Conference on Systems and Informatics (ICSAI). IEEE, 2018: 305-310.
[8] Imran M, Elbassuoni S, Castillo C, et al. Extracting Information Nuggets from Disaster-Related Messages in Social Media[C]// Proceedings of the 10th International Conference on Information Systems for Crisis Response and Management. IEEE, 2013.
[9] Ni J X, Liu X, Zhou Q F, et al. A Knowledge Graph Based Disaster Storyline Generation Framework[C]// Proceedings of the 2019 Chinese Control and Decision Conference (CCDC). IEEE, 2019: 4432-4437.
[10] Chowdhury S R, Imran M, Asghar M R, et al. Tweet4act: Using Incident-specific Profiles for Classifying Crisis-related Messages[C]// Proceedings of the 10th International Conference on Information Systems for Crisis Response and Management. IEEE, 2013.
[11] Basu M, Shandilya A, Khosla P, et al. Extracting Resource Needs and Availabilities from Microblogs for Aiding Post-Disaster Relief Operations[J]. IEEE Transactions on Computational Social Systems, 2019, 6(3):604-618.
doi: 10.1109/TCSS.2019.2914179
[12] 马超. 多模态环境下的突发事件信息组织研究[D]. 武汉: 武汉大学, 2020.
[12] ( Ma Chao. Research on Emergency Information Organization in Multimodal Environment[D]. Wuhan: Wuhan University, 2020.)
[13] Nguyen D T, Ofli F, Imran M, et al. Damage Assessment from Social Media Imagery Data During Disasters[C]// Proceedings of the 2017 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining 2017. ACM, 2017: 569-576.
[14] Cirneanu A L, Popescu D, Ichim L. CNN Based on LBP for Evaluating Natural Disasters[C]// Proceedings of the 15th International Conference on Control, Automation, Robotics and Vision (ICARCV). IEEE, 2018: 568-573.
[15] Weber E, Marzo N, Papadopoulos D P, et al. Detecting Natural Disasters, Damage, and Incidents in the Wild[C]// Proceedings of the 16th European Conference on Computer Vision. IEEE, 2020: 331-350.
[16] Alam F, Ofli F, Imran M. CrisisMMD: Multimodal Twitter Datasets from Natural Disasters[OL]. arXiv Preprint, arXiv:1805.00713.
[17] Ofli F, Alam F, Imran M. Analysis of Social Media Data Using Multimodal Deep Learning for Disaster Response[OL]. arXiv Preprint, arXiv:2004.11838.
[18] 刘宗田, 黄美丽, 周文, 等. 面向事件的本体研究[J]. 计算机科学, 2009, 36(11):189-192,199.
[18] ( Liu Zongtian, Huang Meili, Zhou Wen, et al. Research on Event-oriented Ontology Model[J]. Computer Science, 2009, 36(11):189-192, 199.)
[19] 范维澄, 闪淳昌. 公共安全与应急管理[M]. 北京: 科学出版社, 2017.
[19] ( Fan Weicheng, Shan Chunchang. Public Safety and Emergency Management[M]. Beijing: Science Press, 2017.)
[20] Cooper A. The Inmates are Running the Asylum: Why High-tech Products Drive Us Crazy and How to Restore the Sanity[M]. Indianapolis: SAMS, 2004.
[21] Fohringer J, Dransch D, Kreibich H, et al. Social Media as an Information Source for Rapid Flood Inundation Mapping[J]. Natural Hazards and Earth System Sciences, 2015, 15(12):2725-2738.
[22] Huang G, Liu Z, van der Maaten L, et al. Densely Connected Convolutional Networks[C]// Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 2017: 2261-2269.
[23] Ren S Q, He K M, Girshick R, et al. Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39(6):1137-1149.
doi: 10.1109/TPAMI.2016.2577031
[24] Deng J, Dong W, Socher R, et al. ImageNet: A Large-Scale Hierarchical Image Database[C]// Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 2009: 248-255.
[25] Chen K, Wang J, Pang J, et al. MMDetection: Open MMLab Detection Toolbox and Benchmark[OL]. arXiv Preprint, arXiv:1906.07155.
[26] Lin T Y, Maire M, Belongie S, et al. Microsoft COCO: Common Objects in Context[C]// Proceedings of the 2014 European Conference on Computer Vision. Springer, 2014: 740-755.
[27] Vaswani A, Shazeer N, Parmar N, et al. Attention is All You Need[OL]. arXiv Preprint, arXiv:1706.03762.
[28] Devlin J, Chang M W, Lee K, et al. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding[OL]. arXiv Preprint, arXiv:1810.04805.
[29] Baum E B. On the Capabilities of Multilayer Perceptrons[J]. Journal of Complexity, 1988, 4(3):193-215.
doi: 10.1016/0885-064X(88)90020-9
[30] 李显鑫. 基于图文相关的突发事件图片采集研究[D]. 武汉: 武汉大学, 2020.
[30] ( Li Xianxin. Research on Picture Collection of Emergencies Based on the Correlation Between Pictures and Texts[D]. Wuhan: Wuhan University, 2020.)
[31] Shim H. PHash: A Memory-efficient, High-performance Key-value Store for Large-scale Data-intensive Applications[J]. Journal of Systems and Software, 2017, 123:33-44.
doi: 10.1016/j.jss.2016.09.047
[32] Zadeh A, Chen M H, Poria S, et al. Tensor Fusion Network for Multimodal Sentiment Analysis[C]// Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. 2017: 1103-1114.
[33] 李纲, 张霁, 毛进, 等. 灾害事件下社交媒体图文相关性研究[J]. 情报学报, 2020, 39(11):1223-1231.
[33] ( Li Gang, Zhang Ji, Mao Jin, et al. Relevance Between Image and Text of Social Media Posts for Disasters[J]. Journal of the China Society for Scientific and Technical Information, 2020, 39(11):1223-1231.)
[1] 范涛, 王昊, 李跃艳, 邓三鸿. 基于多模态融合的非遗图片分类研究*[J]. 数据分析与知识发现, 2022, 6(2/3): 329-337.
[2] 安璐, 徐曼婷. 突发公共卫生事件情境下网民对政务微博信任度的测度*[J]. 数据分析与知识发现, 2022, 6(1): 55-68.
[3] 马莹雪,赵吉昌. 自然灾害期间微博平台的舆情特征及演变*——以台风和暴雨数据为例[J]. 数据分析与知识发现, 2021, 5(6): 66-79.
[4] 谢豪,毛进,李纲. 基于多层语义融合的图文信息情感分类研究*[J]. 数据分析与知识发现, 2021, 5(6): 103-114.
[5] 张国标,李洁. 融合多模态内容语义一致性的社交媒体虚假新闻检测*[J]. 数据分析与知识发现, 2021, 5(5): 21-29.
[6] 王雨竹,谢珺,陈波,续欣莹. 基于跨模态上下文感知注意力的多模态情感分析 *[J]. 数据分析与知识发现, 2021, 5(4): 49-59.
[7] 朱路, 邓芳, 刘坤, 贺婷婷, 刘媛媛. 基于语义自编码哈希学习的跨模态检索方法*[J]. 数据分析与知识发现, 2021, 5(12): 110-122.
[8] 程铁军, 王曼, 黄宝凤, 冯兰萍. 基于CEEMDAN-BP模型的突发事件网络舆情预测研究*[J]. 数据分析与知识发现, 2021, 5(11): 59-67.
[9] 尹浩然,曹金璇,曹鲁喆,王国栋. 扩充语义维度的BiGRU-AM突发事件要素识别研究*[J]. 数据分析与知识发现, 2020, 4(9): 91-99.
[10] 李纲, 管为栋, 马亚雪, 毛进. 学术论文的社交媒体可见性预测研究*[J]. 数据分析与知识发现, 2020, 4(8): 63-74.
[11] 刘倩, 李晨亮. 基于社交媒体的话题演变研究综述*[J]. 数据分析与知识发现, 2020, 4(8): 1-14.
[12] 邓建高,张璇,傅柱,韦庆明. 基于系统动力学的突发事件网络舆情传播研究:以“江苏响水爆炸事故”为例*[J]. 数据分析与知识发现, 2020, 4(2/3): 110-121.
[13] 梁艳平,安璐,刘静. 同类突发公共卫生事件微博话题共振研究*[J]. 数据分析与知识发现, 2020, 4(2/3): 122-133.
[14] 谭荧,张进,夏立新. 社交媒体情境下的情感分析研究综述[J]. 数据分析与知识发现, 2020, 4(1): 1-11.
[15] 吴小兰,章成志. 学术社交媒体视角下学科知识流动规律研究*——以科学网为例[J]. 数据分析与知识发现, 2019, 3(4): 107-116.
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
版权所有 © 2015 《数据分析与知识发现》编辑部
地址:北京市海淀区中关村北四环西路33号 邮编:100190
电话/传真:(010)82626611-6626,82624938
E-mail:jishu@mail.las.ac.cn