Please wait a minute...
Data Analysis and Knowledge Discovery  2022, Vol. 6 Issue (12): 23-31    DOI: 10.11925/infotech.2096-3467.2022.0308
Current Issue | Archive | Adv Search |
Detecting Sarcasm from Travel Reviews Based on Cross-Modal Deep Learning
Liu Yang1,2(),Ma Lili3,Zhang Wen1,Hu Zhongyi1,2,Wu Jiang1,2
1School of Information Management, Wuhan University, Wuhan 430072,China
2Center for E-commerce Research and Development, Wuhan University, Wuhan 430072, China
3Economics and Management School, Wuhan University, Wuhan 430072, China
Download: PDF (1632 KB)   HTML ( 28
Export: BibTeX | EndNote (RIS)      

[Objective] Based on the cross-modal deep learning method, this paper analyzes consumers’ sentiments in travel reviews and identifies their sarcastic expression. [Methods] First, we encoded multi-modal information. Then, we extracted the interaction information between texts and pictures with the graph neural network. Finally, we used the attention mechanism to identify multi-modal features and sarcasm. [Results] We examined the proposed model with travel reviews from Yelp. The accuracy of sarcasm detection reached 88.77%, which is better than the baseline models. [Limitations] We only examined the proposed model with reviews on Hilton hotels, which needs to be expanded in the future. [Conclusions] The proposed model could extract interaction information between different modal of data, that effectively improve the accuracy of sarcasm detection.

Key wordsCross-Modal      Deep Learning      Travel Reviews      Sarcasm Detection     
Received: 06 April 2022      Published: 03 February 2023
ZTFLH:  TP391  
Fund:National Key R&D Program of China(2019YFB1405600);National Natural Science Foundation of China(72171183);National Natural Science Foundation of China(72204190)
Corresponding Authors: Liu Yang,ORCID:0000-0002-9410-1755     E-mail:

Cite this article:

Liu Yang, Ma Lili, Zhang Wen, Hu Zhongyi, Wu Jiang. Detecting Sarcasm from Travel Reviews Based on Cross-Modal Deep Learning. Data Analysis and Knowledge Discovery, 2022, 6(12): 23-31.

URL:     OR

Example of Sarcasm in Travel Reviews
Cross-Modal Sarcasm Recognition Model
模态 模型 Accuracy/% Precision/% Recall/% F1-Score/%
文本模态 TextCNN 82.93 83.45 84.65 82.22
Bi-LSTM 84.15 84.26 84.17 85.58
BERT 82.14 81.33 82.45 82.68
SVM+BERT 84.98 85.43 84.79 85.25
图像模态 ResNet 72.55 74.88 73.64 71.09
EfficientNet 78.02 77.23 75.15 76.34
UPB-MTL 79.41 78.43 78.92 78.45
多模态 BERT+ResNet 84.75 84.02 84.65 83.82
Attention+BERT+ResNet 86.46 84.18 85.32 85.93
本文模型 88.77 86.34 87.42 87.86
Algorithm Performance Comparison Results
The Results of Different Models
模态 模型 Accuracy/% Precision/% Recall/% F1-score/%

-Attention 86.85 85.13 85.80 86.56
-GCN 83.29 84.54 83.48 84.07
本文 88.77 86.34 87.42 87.86
Results of Ablation Experiments
游客ID 时间 评论文本 图片 正确
Alex P 2021.1.29 We stayed at Hilton Palace for one night while we stopped and enjoyed Disney Springs while we drove through Florida. For the price we paid, I'm disappointed. The room was tiny, outdated, sliding window was must have had the gases released through the window panes that it fogged it completely. Slept on a mattress cover and not sheets. which was weird. I wouldn't stay here again. I guess I expected more out of Hilton across from Disney World.

Christina Z 2021.8.6 I was here for work. There was a convention at the Orange County Convention Center. The hotel was very clean. Covid rules were very relaxed, but I expected that in Florida. Their room service is still suspended but their restaurant David's does take out. The staff was extremely friendly. Their WiFi was awful in the rooms and made it challenging to do work. Likewise the TV would keep going in and out. We weren't having any storms so I'm not sure what the problem was. I would stay here again. I just hope they fix their internet

Misclassified Travel Reviews
[1] 陈晓红, 李杨扬, 宋丽洁, 等. 数字经济理论体系与研究展望[J]. 管理世界, 2022, 38(2): 208-224.
[1] (Chen Xiaohong, Li Yangyang, Song Lijie, et al. Theoretical Framework and Research Prospect of Digital Economy[J]. Journal of Management World, 2022, 38(2): 208-224.)
[2] Alarcón-del-Amo M D C, Rialp-Criado A, Rialp-Criado J. Examining the Impact of Managerial Involvement with Social Media on Exporting Firm Performance[J]. International Business Review, 2018, 27(2): 355-366.
doi: 10.1016/j.ibusrev.2017.09.003
[3] Goldin P R, McRae K, Ramel W, et al. The Neural Bases of Emotion Regulation: Reappraisal and Suppression of Negative Emotion[J]. Biological Psychiatry, 2008, 63(6): 577-586.
doi: 10.1016/j.biopsych.2007.05.031 pmid: 17888411
[4] Braunstein L M, Gross J J, Ochsner K N. Explicit and Implicit Emotion Regulation: A Multi-Level Framework[J]. Social Cognitive and Affective Neuroscience, 2017, 12(10): 1545-1557.
doi: 10.1093/scan/nsx096 pmid: 28981910
[5] Joshi A, Bhattacharyya P, Carman M J. Automatic Sarcasm Detection: A Survey[J]. ACM Computing Surveys, 2017, 50(5): 1-22.
[6] Gibbs R W. On the Psycholinguistics of Sarcasm[J]. Journal of Experimental Psychology: General, 1986, 115(1): 3-15.
doi: 10.1037/0096-3445.115.1.3
[7] Ordenes F V, Ludwig S, de Ruyter K, et al. Unveiling What is Written in the Stars: Analyzing Explicit, Implicit, and Discourse Patterns of Sentiment in Social Media[J]. Journal of Consumer Research, 2017, 43(6): 875-894.
[8] Eke C I, Norman A A, Shuib L, et al. Sarcasm Identification in Textual Data: Systematic Review, Research Challenges and Open Directions[J]. Artificial Intelligence Review, 2020, 53(6): 4215-4258.
doi: 10.1007/s10462-019-09791-8
[9] Riloff E, Qadir A, Surve P, et al. Sarcasm as Contrast Between a Positive Sentiment and Negative Situation[C]// Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing. 2013: 704-714.
[10] Bharti S K, Babu K S, Jena S K. Parsing-Based Sarcasm Sentiment Recognition in Twitter Data[C]// Proceedings of 2015 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining. 2015: 1373-1380.
[11] Bing L. Sentiment Analysis and Opinion Mining[J]. Synthesis Lectures on Human Language Technologies, 2012, 5(1): 1-167.
[12] Bharti S K, Pradhan R, Babu K S, et al. Sarcasm Analysis on Twitter Data Using Machine Learning Approaches[J]. Trends in Social Network Analysis, 2017: 51-76.
[13] González-Ibánez R, Muresan S, Wacholder N. Identifying Sarcasm in Twitter: A Closer Look[C]// Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics:Human Language Technologies. 2011: 581-586.
[14] Zhang Y Z, Liu Y C, Li Q C, et al. CFN: A Complex-Valued Fuzzy Network for Sarcasm Detection in Conversations[J]. IEEE Transactions on Fuzzy Systems, 2021, 29(12): 3696-3710.
doi: 10.1109/TFUZZ.2021.3072492
[15] Ghosh A, Veale D T. Fracking Sarcasm Using Neural Network[C]// Proceedings of the 7th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis. 2016: 161-169.
[16] McFee B, Lanckriet G, Jebara T. Learning Multi-modal Similarity[J]. Journal of Machine Learning Research, 2011, 12: 491-523.
[17] Baltrušaitis T, Ahuja C, Morency L P. Multimodal Machine Learning: A Survey and Taxonomy[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019, 41(2): 423-443.
doi: 10.1109/TPAMI.2018.2798607 pmid: 29994351
[18] 刘金桥, 吴金强. 机器视觉系统发展及其应用[J]. 机械工程与自动化, 2010(1): 215-216.
[18] (Liu Jinqiao, Wu Jinqiang. Development of Machine Vision System and Its Application[J]. Mechanical Engineering & Automation, 2010(1): 215-216.)
[19] Law R, Li G, Fong D K C, et al. Tourism Demand Forecasting: A Deep Learning Approach[J]. Annals of Tourism Research, 2019, 75: 410-423.
doi: 10.1016/j.annals.2019.01.014
[20] Sun S L, Wei Y J, Tsui K L, et al. Forecasting Tourist Arrivals with Machine Learning and Internet Search Index[J]. Tourism Management, 2019, 70: 1-10.
doi: 10.1016/j.tourman.2018.07.010
[21] Yang Y, Tang J Y, Luo H, et al. Hotel Location Evaluation: A Combination of Machine Learning Tools and Web GIS[J]. International Journal of Hospitality Management, 2015, 47: 14-24.
doi: 10.1016/j.ijhm.2015.02.008
[22] Deng N, Li X. Feeling a Destination Through the “Right” Photos: A Machine Learning Model for DMOs’ Photo Selection[J]. Tourism Management, 2018, 65: 267-278.
doi: 10.1016/j.tourman.2017.09.010
[23] O’Connor P. Managing a Hotel’s Image on TripAdvisor[J]. Journal of Hospitality Marketing & Management, 2010, 19(7): 754-772.
[24] Ma Y F, Xiang Z, Du Q Z, et al. Effects of User-Provided Photos on Hotel Review Helpfulness: An Analytical Approach with Deep Leaning[J]. International Journal of Hospitality Management, 2018, 71: 120-131.
doi: 10.1016/j.ijhm.2017.12.008
[25] Tenney I, Das D, Pavlick E. BERT Rediscovers the Classical NLP Pipeline[OL]. arXiv Preprint, arXiv:1905.05950.
[26] Huang Z, Xu W, Yu K. Bidirectional LSTM-CRF Models for Sequence Tagging[OL]. arXiv Preprint, arXiv:1508.01991.
[27] He K M, Zhang X Y, Ren S Q, et al. Identity Mappings in Deep Residual Networks[C]// Proceedings of European Conference on Computer Vision. 2016: 630-645.
[28] Scarselli F, Gori M, Tsoi A C, et al. The Graph Neural Network Model[J]. IEEE Transactions on Neural Networks, 2009, 20(1): 61-80.
doi: 10.1109/TNN.2008.2005605 pmid: 19068426
[29] Shaw P, Uszkoreit J, Vaswani A. Self-attention with Relative Position Representations[OL]. arXiv Preprint, arXiv:1803.02155.
[30] Zhang Y, Wallace B. A Sensitivity Analysis of (and Practitioners’ Guide to) Convolutional Neural Networks for Sentence Classification[OL]. arXiv Preprint, arXiv:1510.03820.
[31] Tan M X, Le Q V. EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks[C]// Proceedings of the 36th International Conference on Machine Learning. 2019: 6105-6114.
[32] Vlad G A, Zaharia G E, Cercel D C, et al. UPB at SemEval-2020 Task 8: Joint Textual and Visual Modeling in a Multi-task Learning Architecture for Memotion Analysis[OL]. arXiv Preprint, arXiv:2009.02779.
[33] Pan H L, Lin Z, Fu P, et al. Modeling Intra and Inter-modality Incongruity for Multi-modal Sarcasm Detection[C]// Findings of the Association for Computational Linguistics:EMNLP 2020. 2020: 1383-1392.
[1] Cheng Quan, She Dexin. Drug Recommendation Based on Graph Neural Network with Patient Signs and Medication Data[J]. 数据分析与知识发现, 2022, 6(9): 113-124.
[2] Chen Yuanyuan, Ma Jing. Detecting Multimodal Sarcasm Based on SC-Attention Mechanism[J]. 数据分析与知识发现, 2022, 6(9): 40-51.
[3] Wang Lu, Le Xiaoqiu. Research Progress on Citation Analysis of Scientific Papers[J]. 数据分析与知识发现, 2022, 6(4): 1-15.
[4] Zheng Xiao, Li Shuqing, Zhang Zhiwang. Measuring User Item Quality with Rating Analysis for Deep Recommendation Model[J]. 数据分析与知识发现, 2022, 6(4): 39-48.
[5] Yu Chuanming, Lin Hongjun, Zhang Zhengang. Joint Extraction Model for Entities and Events with Multi-task Deep Learning[J]. 数据分析与知识发现, 2022, 6(2/3): 117-128.
[6] Zhang Yunqiu, Li Bocheng, Chen Yan. Automatic Classification with Unbalanced Data for Electronic Medical Records[J]. 数据分析与知识发现, 2022, 6(2/3): 233-241.
[7] Zhang Fangcong, Qin Qiuli, Jiang Yong, Zhuang Runtao. Named Entity Recognition for Chinese EMR with RoBERTa-WWM-BiLSTM-CRF[J]. 数据分析与知识发现, 2022, 6(2/3): 251-262.
[8] Hu Yamin, Wu Xiaoyan, Chen Fang. Review of Technology Term Recognition Studies Based on Machine Learning[J]. 数据分析与知识发现, 2022, 6(2/3): 7-17.
[9] Cao Lina,Zhang Jian,Chen Jindong,Fan Hui. Comprehensive Quality Profiling for Micro-, Small-, and Medium-sized Enterprises Based on Deep Learning[J]. 数据分析与知识发现, 2022, 6(11): 126-138.
[10] Li Zhi, Sun Rui, Yao Yuxuan, Li Xiaohuan. Recommending Point-of-Interests with Real-Time Event Detection[J]. 数据分析与知识发现, 2022, 6(10): 114-127.
[11] Huang Xuejian, Liu Yuyang, Ma Tinghuai. Classification Model for Scholarly Articles Based on Improved Graph Neural Network[J]. 数据分析与知识发现, 2022, 6(10): 93-102.
[12] Zhou Zeyu,Wang Hao,Zhao Zibo,Li Yueyan,Zhang Xiaoqin. Construction and Application of GCN Model for Text Classification with Associated Information[J]. 数据分析与知识发现, 2021, 5(9): 31-41.
[13] Xu Yuemei, Wang Zihou, Wu Zixin. Predicting Stock Trends with CNN-BiLSTM Based Multi-Feature Integration Model[J]. 数据分析与知识发现, 2021, 5(7): 126-138.
[14] Zhao Danning,Mu Dongmei,Bai Sen. Automatically Extracting Structural Elements of Sci-Tech Literature Abstracts Based on Deep Learning[J]. 数据分析与知识发现, 2021, 5(7): 70-80.
[15] Huang Mingxuan,Jiang Caoqing,Lu Shoudong. Expanding Queries Based on Word Embedding and Expansion Terms[J]. 数据分析与知识发现, 2021, 5(6): 115-125.
  Copyright © 2016 Data Analysis and Knowledge Discovery   Tel/Fax:(010)82626611-6626,82624938