Please wait a minute...
Data Analysis and Knowledge Discovery  2020, Vol. 4 Issue (12): 85-94    DOI: 10.11925/infotech.2096-3467.2020.0535
Current Issue | Archive | Adv Search |
Sentiment Analysis of Cross-Domain Product Reviews Based on Feature Fusion and Attention Mechanism
Qi Ruihua1,2(),Jian Yue1,2,Guo Xu2,Guan Jinghua2,Yang Mingxin1,2
1Research Center for Language Intelligence, Dalian University of Foreign Languages, Dalian 116044, China
2School of Software Engineering, Dalian University of Foreign Languages, Dalian 116044, China
Download: PDF (1050 KB)   HTML ( 13
Export: BibTeX | EndNote (RIS)      
Abstract  

[Objective] This paper tries to address the issues of labelled data shortage, aiming to distinguish the weights of sentiment characteristics in cross-domain sentiment classification. [Methods] We proposed a sentiment classification model for cross-domain product reviews based on feature fusion representation and the attention mechanism. First, this model integrated Bert and cross-domain word vectors to generate cross-domain unified feature space. Then, it extracted the weights of global and local features through attention mechanism. [Results] We examined our model with public review data from Amazon and found the average accuracy of the proposed model was up-to 95.93%, which was 9.33% higher than the existing model. [Limitations] More research is needed to evaluate our model with large-scale multi-domain data sets. [Conclusions] The proposed model could effectively analyze sentiment information.

Key wordsFeature Fusion      Attention Mechanism      Cross-Domain      Sentiment Classification     
Received: 08 June 2020      Published: 25 December 2020
ZTFLH:  TP393  
Corresponding Authors: Qi Ruihua     E-mail: rhqi@dlufl.edu.cn

Cite this article:

Qi Ruihua,Jian Yue,Guo Xu,Guan Jinghua,Yang Mingxin. Sentiment Analysis of Cross-Domain Product Reviews Based on Feature Fusion and Attention Mechanism. Data Analysis and Knowledge Discovery, 2020, 4(12): 85-94.

URL:

http://manu44.magtech.com.cn/Jwk_infotech_wk3/EN/10.11925/infotech.2096-3467.2020.0535     OR     http://manu44.magtech.com.cn/Jwk_infotech_wk3/EN/Y2020/V4/I12/85

Sentiment Classification Model for Cross-Domain Product Reviews Based on Feature Fusion and Attention Mechanism
领域 积极评论 消极评论 无标注文本
Books 3 000 3 000 9 750
DVD disk 3 000 3 000 11 843
Electronics 3 000 3 000 17 009
Kitchen appliances 3 000 3 000 13 856
Videos 3 000 3 000 30 180
Experimental Data Set
参数名称 参数名称
最大长度 120 优化器 Adam
词向量维度 768,1 068,
1 068
损失函数 binary_
crossentropy
LSTM隐藏单元 128 batch_size 32
全连接层1 128, activation=“tanh” 全连接层2 2, activation=“softmax”
Dropout 0.5 输出层激活函数 Softmax
Experimental Parameters of Cross-Domain Sentiment Classification
源领域 目标领域 S-only SFA DANN mSDA HATN CDSA-B CDSA-F CDSA-F-Att
Book DVD 80.57% 82.85% 83.42% 86.12% 87.07% 93.20% 94.96% 95.46%
Book Electronic 73.65% 76.38% 76.27% 79.02% 85.75% 93.68% 95.46% 95.43%
Book Kitchen 71.63% 78.10% 77.90% 81.05% 87.03% 94.65% 95.20% 96.11%
Book Video 81.45% 82.95% 83.23% 84.98% 87.80% 95.56% 96.10% 96.53%
DVD Book 76.45% 80.20% 80.77% 85.17% 87.78% 93.41% 94.55% 94.86%
DVD Electronic 73.12% 76.00% 76.35% 76.17% 86.32% 92.86% 95.08% 95.35%
DVD Kitchen 73.43% 77.50% 78.15% 82.60% 87.47% 94.76% 96.28% 96.51%
DVD Video 82.75% 85.95% 85.95% 83.80% 89.12% 96.15% 96.65% 97.10%
Electronic Book 68.87% 72.35% 73.53% 79.92% 84.03% 93.15% 94.70% 94.86%
Electronic DVD 72.60% 75.93% 76.27% 82.63% 84.32% 93.38% 95.68% 96.13%
Electronic Kitchen 84.63% 86.50% 84.53% 85.80% 90.08% 95.70% 96.41% 96.60%
Electronic Video 72.48% 75.65% 77.20% 81.70% 84.08% 95.48% 96.23% 96.71%
Kitchen Book 71.53% 73.97% 74.17% 80.55% 84.88% 93.68% 94.03% 95.00%
Kitchen DVD 73.32% 75.67% 75.32% 82.18% 84.72% 93.40% 95.81% 96.03%
Kitchen Electronic 83.15% 85.38% 85.53% 88.00% 89.33% 94.63% 96.00% 96.01%
Kitchen Video 76.08% 77.97% 76.37% 81.47% 84.85% 95.90% 96.91% 96.80%
Video Book 77.03% 79.48% 80.03% 83.00% 87.10% 93.88% 94.18% 94.19%
Video DVD 82.43% 83.65% 84.15% 85.90% 87.90% 95.30% 95.96% 96.50%
Video Electronic 71.87% 75.93% 75.72% 77.67% 85.98% 94.00% 94.98% 95.86%
Video Kitchen 71.33% 74.78% 75.22% 79.52% 86.45% 95.80% 96.28% 96.71%
平均准确率 75.92% 78.69% 79.00% 82.36% 86.60% 94.42% 95.57% 95.93%
Experimental Accuracy on Amazon Review Data Set
Average Accuracy of Eight Models in Source-Target Domain Experiments
Average Accuracy of CDSA-F Model in Source-Target Domain Experiments
Average Accuracy of CDSA-F-Att Model in Source-Target Domain Experiments
[1] Al-Moslmi T, Omar N, Abdullah S , et al. Approaches to Cross-Domain Sentiment Analysis: A Systematic Literature Review[J]. IEEE Access, 2017,5:16173-16192.
[2] Lai S W, Liu K, He S Z , et al. How to Generate a Good Word Embedding[J]. IEEE Intelligent Systems, 2016,31(6):5-14.
[3] Weiss K, Khoshgoftaar T M, Wang D D . A Survey of Transfer Learning[J]. Journal of Big Data, 2016, 3(1): Article No. 9.
[4] Tahmoresnezhad J, Hashemi S . Visual Domain Adaptation via Transfer Feature Learning[J]. Knowledge and Information Systems, 2017,50(2):585-605.
[5] Sun M, Tan Q, Ding R W, et al. Cross-Domain Sentiment Classification Using Deep Learning Approach [C]//Proceedings of the 3rd International Conference on Cloud Computing and Intelligence Systems. IEEE, 2014: 60-64.
[6] 余传明 . 基于深度循环神经网络的跨领域文本情感分析[J]. 图书情报工作, 2018,62(11):23-34.
[6] ( Yu Chuanming . A Cross-Domain Text Sentiment Analysis Based on Deep Recurrent Neural Network[J]. Library and Information Service, 2018,62(11):23-34.)
[7] Ganin Y, Ustinova E, Ajakan H , et al. Domain-Adversarial Training of Neural Networks[J]. The Journal of Machine Learning Research, 2016,17(1):2096-2130.
[8] Gamon M, Aue A. Automatic Identification of Sentiment Vocabulary: Exploiting Low Association with Known Sentiment Terms [C]//Proceedings of the ACL Workshop on Feature Engineering for Machine Learning in Natural Language Processing. 2005: 57-64.
[9] Dai W Y, Yang Q, Xue G R, et al. Boosting for Transfer Learning [C]//Proceedings of the 24th International Conference on Machine Learning. 2007: 193-200.
[10] Zhang S W, Liu H L, Yang L, et al. A Cross-domain Sentiment Classification Method Based on Extraction of Key Sentiment Sentence [C]//Proceedings of the 4th CCF Conference on Natural Language Processing and Chinese Computing-Volume 9362. 2015: 90-101.
[11] Blitzer J, McDonald R, Pereira F. Domain Adaptation with Structural Correspondence Learning [C]//Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing. 2006: 120-128.
[12] Blitzer J, Dredze M, Pereira F. Biographies, Bollywood, Boom-Boxes and Blenders: Domain Adaptation for Sentiment Classification [C]//Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics. 2007: 440-447.
[13] Pan S J, Ni X, Sun J T, et al. Cross-Domain Sentiment Classification via Spectral Feature Alignment [C]// Proceedings of the 19th International Conference on World Wide Web. 2010: 751-760.
[14] Wei X C, Lin H F, Yang L . Cross-Domain Sentiment Classification via Constructing Semantic Correlation[J]. IAENG International Journal of Computer Science, 2017,44(2):172-179.
[15] Glorot X, Bordes A, Bengio Y. Domain Adaptation for Large-Scale Sentiment Classification: A Deep Learning Approach [C]//Proceedings of the 28th International Conference on Machine Learning. 2011: 513-520.
[16] Chen M M, Xu Z X, Weinberger K Q, et al. Marginalized Denoising Autoencoders for Domain Adaptation [C]//Proceedings of the 29th International Conference on Machine Learning. 2012: 1627-1634.
[17] Mikolov T, Yih W T, Zweig G. Linguistic Regularities in Continuous Space Word Representations [C]//Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2013: 746-751.
[18] Akhtar M S, Kumar A, Ekbal A, et al. A Hybrid Deep Learning Architecture for Sentiment Analysis [C]// Proceedings of the 26th International Conference on Computational Linguistics: Technical Papers. 2016: 482-493.
[19] Yu J F, Jiang J. Learning Sentence Embeddings with Auxiliary Tasks for Cross-Domain Sentiment Classification [C]//Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. 2016: 236-246.
[20] Li Y T, Baldwin T, Cohn T. What’s in a Domain? Learning Domain-Robust Text Representations Using Adversarial Training [C]//Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2. 2018: 474-479.
[21] Zhang L W, Tu K W, Zhang Y. Latent Variable Sentiment Grammar [C]//Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. 2019: 4642-4651.
[22] Tai K S, Socher R, Manning C D. Improved Semantic Representations from Tree-Structured Long Short-Term Memory Networks [C]//Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1). 2015: 1556-1566.
[23] Qian Q, Huang M L, Lei J H, et al. Linguistically Regularized LSTM for Sentiment Classification [C]// Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1). 2017: 1679-1689.
[24] Wei X C, Lin H F, Yang L , et al. A Convolution-LSTM-Based Deep Neural Network for Cross-Domain MOOC Forum Post Classification[J]. Information, 2017,8(3). DOI: 10.3390/info8030092.
[25] Raffel C, Ellis D P W . Feed-Forward Networks with Attention can Solve Some Long-Term Memory Problems[OL]. arXiv Preprint, arXiv: 1512.08756.
[26] Yang Z C, Yang D Y, Dyer C, et al. Hierarchical Attention Networks for Document Classification [C]//Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2016: 1480-1489.
[27] Gu Y, Yang K N, Fu S Y, et al. Multimodal Affective Analysis Using Hierarchical Attention Strategy with Word-Level Alignment [C]//Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1). 2018: 2225-2235.
[28] Li Z, Zhang Y, Wei Y, et al. End-to-End Adversarial Memory Network for Cross-Domain Sentiment Classification [C]//Proceedings of the 26th International Joint Conference on Artificial Intelligence. 2017.
[29] Li Z, Wei Y, Zhang Y, et al. Hierarchical Attention Transfer Network for Cross-Domain Sentiment Classification [C]//Proceedings of the 32nd AAAI Conference on Artificial Intelligence. 2018: 5852-5859.
[30] Ben-David S, Blitzer J, Crammer K , et al. A Theory of Learning from Different Domains[J]. Machine Learning, 2010,79(1-2):151-175.
doi: 10.1007/s10994-009-5152-4
[31] Jatnika D, Bijaksana M A, Suryani A A . Word2Vec Model Analysis for Semantic Similarities in English Words[J]. Procedia Computer Science, 2019,157:160-167.
doi: 10.1016/j.procs.2019.08.153
[32] Hochreiter S, Schmidhuber J . Long Short-Term Memory[J]. Neural Computation, 1997,9(8):1735-1780.
doi: 10.1162/neco.1997.9.8.1735 pmid: 9377276
[33] Zhou P, Shi W, Tian J, et al. Attention-Based Bidirectional Long Short-Term Memory Networks for Relation Classification [C]//Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2). 2016: 207-212.
[34] Buja A, Stuetzle W, Shen Y. Loss Functions for Binary Class Probability Estimation and Classification: Structure and Applications[EB/OL]. [ 2020- 07- 12]. http://www-stat.wharton.upenn.edu/~buja/.
[1] Huang Lu,Zhou Enguo,Li Daifeng. Text Representation Learning Model Based on Attention Mechanism with Task-specific Information[J]. 数据分析与知识发现, 2020, 4(9): 111-122.
[2] Yin Haoran,Cao Jinxuan,Cao Luzhe,Wang Guodong. Identifying Emergency Elements Based on BiGRU-AM Model with Extended Semantic Dimension[J]. 数据分析与知识发现, 2020, 4(9): 91-99.
[3] Shi Lei,Wang Yi,Cheng Ying,Wei Ruibin. Review of Attention Mechanism in Natural Language Processing[J]. 数据分析与知识发现, 2020, 4(5): 1-14.
[4] Li Junlian,Wu Yingjie,Deng Panpan,Leng Fuhai. Automatic Data Processing Strategy of Citation Anomie Based on Feature Fusion[J]. 数据分析与知识发现, 2020, 4(5): 38-45.
[5] Xue Fuliang,Liu Lifang. Fine-Grained Sentiment Analysis with CRF and ATAE-LSTM[J]. 数据分析与知识发现, 2020, 4(2/3): 207-213.
[6] Xu Tongtong,Sun Huazhi,Ma Chunmei,Jiang Lifen,Liu Yichen. Classification Model for Few-shot Texts Based on Bi-directional Long-term Attention Features[J]. 数据分析与知识发现, 2020, 4(10): 113-123.
[7] Na Ma,Zhixiong Zhang,Pengmin Wu. Automatic Identification of Term Citation Object with Feature Fusion[J]. 数据分析与知识发现, 2020, 4(1): 89-98.
[8] Yuemin Wu,Ganggui Ding,Bin Hu. Extracting Relationship of Agricultural Financial Texts with Attention Mechanism[J]. 数据分析与知识发现, 2019, 3(5): 86-92.
[9] Qingqing Zhang,Xingshi He,Huimin Wang,Shengjun Meng. Text Sentiment Classification Based on Deep Belief Network[J]. 数据分析与知识发现, 2019, 3(4): 71-79.
[10] Jiaxin Ye,Huixiang Xiong. Recommending Personalized Contents from Cross-Domain Resources Based on Tags[J]. 数据分析与知识发现, 2019, 3(2): 21-32.
[11] Qiang Lu,Zhenfang Zhu,Fuyong Xu,Qiangqiang Guo. Chinese Sentiment Classification Method with Bi-LSTM and Grammar Rules[J]. 数据分析与知识发现, 2019, 3(11): 99-107.
[12] Yuman Li,Zhibo Chen,Fu Xu. Classifying Texts with KACC Model[J]. 数据分析与知识发现, 2019, 3(10): 89-97.
[13] Hui Li,Yaqing Chai. Fine-Grained Sentiment Analysis Based on Convolutional Neural Network[J]. 数据分析与知识发现, 2019, 3(1): 95-103.
[14] Li Xiangdong,Gao Fan,Li Youhai. Categorizing Documents Automatically within Common Semantic Space[J]. 数据分析与知识发现, 2018, 2(9): 66-73.
[15] Wang Shuyi,Liao Huatao,Wu Chake. Mining News on Competitors with Sentiment Classification[J]. 数据分析与知识发现, 2018, 2(3): 70-78.
  Copyright © 2016 Data Analysis and Knowledge Discovery   Tel/Fax:(010)82626611-6626,82624938   E-mail:jishu@mail.las.ac.cn