A Deep Reinforcement Learning Recommendation Model with Multi-modal Features
Pan Huali1,Xie Jun1(),Gao Jing1,Xu Xinying2,Wang Changzheng3
1College of Information and Computer, Taiyuan University of Technology, Jinzhong 030600,China 2College of Electrical and Power Engineering, Taiyuan University of Technology, Taiyuan 030024,China 3Shanxi Tongfang Knowledge Network Digital Publishing Technology Co., Ltd., Taiyuan 030000, China
[Objective] This paper addresses data sparsity and dynamic changes in user interests with multimodal feature fusion and deep reinforcement learning. [Methods] First, we used a pre-trained model and attention mechanism to achieve intra-modal representation and fusion of three modalities. Then, we created a model for user-item interactions. Finally, we utilized the deep reinforcement learning algorithm to capture user interest drift and long and short-term rewards in real time to achieve personalized recommendations. [Results] Compared with the highest value in the baseline models, the proposed model improved precision@5 by 11.8%, 16.5%, 11.4%, and NDCG@5 by 5.3%, 8.0%, 6.4%, on the MovieLens-1M, MovieLens-100K, and Douban datasets, respectively. [Limitations] The user interaction history in the Douban dataset is relatively small, and the proposed model cannot learn more accurate user preferences during training. Compared with the experiments on the MovieLens dataset, we received limited recommendation results. [Conclusions] The proposed model integrates multimodal information to reconstruct the state representation network of deep reinforcement learning, improving the recommendation effect.
潘华莉, 谢珺, 高婧, 续欣莹, 王长征. 融合多模态特征的深度强化学习推荐模型*[J]. 数据分析与知识发现, 2023, 7(4): 114-128.
Pan Huali, Xie Jun, Gao Jing, Xu Xinying, Wang Changzheng. A Deep Reinforcement Learning Recommendation Model with Multi-modal Features. Data Analysis and Knowledge Discovery, 2023, 7(4): 114-128.
(China Internet Network Information Center. Statistical Report of the 49rd Chinese Internet Development[R/OL]. [2023-03-06]. https://www.cnnic.net.cn/NMediaFile/old_attach/P020220721404263787858.pdf. )
[2]
Linden G, Smith B, York J. Amazon.com Recommendations: Item-to-Item Collaborative Filtering[J]. IEEE Internet Computing, 2003, 7(1): 76-80.
doi: 10.1109/MIC.2003.1167344
[3]
Rendle S, Freudenthaler C, Gantner Z, et al. BPR: Bayesian Personalized Ranking from Implicit Feedback[C]// Proceedings of the 25th Conference on Uncertainty in Artificial Intelligence. 2009: 452-461.
[4]
Cheng H T, Koc L, Harmsen J, et al. Wide&Deep Learning for Recommender Systems[C]// Proceedings of the 1st Workshop on Deep Learning for Recommender Systems. 2016: 7-10.
[5]
He X N, He Z K, Song J K, et al. NAIS: Neural Attentive Item Similarity Model for Recommendation[J]. IEEE Transactions on Knowledge and Data Engineering, 2018, 30(12): 2354-2366.
doi: 10.1109/TKDE.2018.2831682
(Yu Li, Du Qihan, Yue Boyan, et al. Survey of Reinforcement Learning Based Recommender Systems[J]. Computer Science, 2021, 48(10): 1-18.)
doi: 10.11896/jsjkx.210200085
[7]
Maillard O A, Munos R, Ryabko D. Selecting the State-Representation in Reinforcement Learning[C]// Proceedings of the 24th International Conference on Neural Information Processing Systems. 2011: 2627-2635.
[8]
Gao J, Li P, Chen Z K, et al. A Survey on Deep Learning for Multimodal Data Fusion[J]. Neural Computation, 2020, 32(5): 829-864.
doi: 10.1162/neco_a_01273
pmid: 32186998
[9]
Zheng G J, Zhang F Z, Zheng Z H, et al. DRN: A Deep Reinforcement Learning Framework for News Recommendation[C]// Proceedings of the 2018 World Wide Web Conference. 2018: 167-176.
[10]
Zhao X Y, Zhang L, Ding Z Y, et al. Recommendations with Negative Feedback via Pairwise Deep Reinforcement Learning[C]// Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 2018: 1040-1048.
[11]
Zhang Y Y, Su X Y, Liu Y. A Novel Movie Recommendation System Based on Deep Reinforcement Learning with Prioritized Experience Replay[C]// Proceedings of 2019 IEEE 19th International Conference on Communication Technology. 2019: 1496-1500.
(Yan Shihong, Ma Weizhi, Zhang Min, et al. Reinforcement Learning with User Long-Term and Short-Term Preference for Personalized Recommendation[J]. Journal of Chinese Information Processing, 2021, 35(8): 107-116.)
[13]
Zhao X Y, Xia L, Zhang L, et al. Deep Reinforcement Learning for Page-Wise Recommendations[C]// Proceedings of the 12th ACM Conference on Recommender Systems. 2018: 95-103.
[14]
Zhou Q L. A Novel Movies Recommendation Algorithm Based on Reinforcement Learning with DDPG Policy[J]. International Journal of Intelligent Computing and Cybernetics, 2020, 13(1): 67-79.
doi: 10.1108/IJICC-09-2019-0103
[15]
Liu F, Tang R M, Li X T, et al. State Representation Modeling for Deep Reinforcement Learning Based Recommendation[J]. Knowledge-Based Systems, 2020, 205: 106170.
doi: 10.1016/j.knosys.2020.106170
[16]
Xin X, Karatzoglou A, Arapakis I, et al. Supervised Advantage Actor-Critic for Recommender Systems[C]// Proceedings of the 15th ACM International Conference on Web Search and Data Mining. 2022: 1186-1196.
[17]
He R N, McAuley J. VBPR: Visual Bayesian Personalized Ranking from Implicit Feedback[C]// Proceedings of the 30th AAAI Conference on Artificial Intelligence. 2016: 144-150.
[18]
Kim D, Park C, Oh J, et al. Convolutional Matrix Factorization for Document Context-Aware Recommendation[C]// Proceedings of the 10th ACM Conference on Recommender Systems. 2016: 233-240.
(Ma Yingxue, Gan Mingxin, Xiao Kejun. A Matrix Factorization Recommendation Method with Tags and Contents[J]. Data Analysis and Knowledge Discovery, 2021, 5(5): 71-82.)
[20]
Mangolin R B, Pereira R M, Jr Britto A S, et al. A Multimodal Approach for Multi-Label Movie Genre Classification[J]. Multimedia Tools and Applications, 2022, 81(14): 19071-19096.
doi: 10.1007/s11042-020-10086-2
[21]
Wang J H, Wu Y T, Wang L. Predicting Implicit User Preferences with Multimodal Feature Fusion for Similar User Recommendation in Social Media[J]. Applied Sciences, 2021, 11(3): 1064.
doi: 10.3390/app11031064
[22]
Wang Z, Chen H L, Li Z, et al. VRConvMF: Visual Recurrent Convolutional Matrix Factorization for Movie Recommendation[J]. IEEE Transactions on Emerging Topics in Computational Intelligence, 2022, 6(3): 519-529..
doi: 10.1109/TETCI.2021.3102619
(Han Tengyue, Niu Shaozhang, Zhang Wen. Multimodal Sequential Recommendation Algorithm Based on Contrastive Learning[J]. Journal of Computer Applications, 2022, 42(6): 1683-1688.)
doi: 10.11772/j.issn.1001-9081.2021081417
[24]
Mikolov T, Sutskever I, Chen K, et al. Distributed Representations of Words and Phrases and Their Compositionality[C]// Proceedings of the 26th International Conference on Neural Information Processing Systems - Volume 2. 2013: 3111-3119.
[25]
Lillicrap T P, Hunt J J, Pritzel A, et al. Continuous Control with Deep Reinforcement Learning[OL]. arXiv Preprint, arXiv:1509.02971.
[26]
Mnih V, Kavukcuoglu K, Silver D, et al. Human-Level Control Through Deep Reinforcement Learning[J]. Nature, 2015, 518(7540): 529-533.
doi: 10.1038/nature14236
