Please wait a minute...
Data Analysis and Knowledge Discovery  2022, Vol. 6 Issue (12): 123-134    DOI: 10.11925/infotech.2096-3467.2022.0106
Current Issue | Archive | Adv Search |
Predicting Stock Prices Based on RoBERTa-TCN and Sentimental Characteristics
Yan Dongmei(),He Wenxin,Chen Zhi
School of Science and Engineering, Tianjin University of Finance and Economics, Tianjin 300222, China
Download: PDF (957 KB)   HTML ( 18 ) 
Export: BibTeX | EndNote (RIS)      
Abstract  

[Objective] Thisf paper aims to improve the prediction of stock prices with the help of investors’sentimental characteristics. [Methods] First, we constructed investors’ sentimental characteristics with the RoBERTa model and extracted the stock price characteristics with the TCN network. Then, we used the attention mechanism to merge these characteristics. Finally, we constructed the new RoBERTa-TCN model for stock price prediction. [Results] Compared with the experimental results of three models LSTM, GRU and TCN on six stock datasets, RoBERTa-TCN model has an average improvement of about 0.4906 on four different evaluation indicators. [Limitations] We did not examine the impacts of trading dates on the stock prices. [Conclusions] The RoBERTa-TCN model could effectively predict stock prices.
Key words： Stock Prediction      Sentimental Characteristics      RoBERTa      Temporal Convolutional Neural Network      Attention Mechanism     
Received: 11 February 2022      Published: 03 February 2023
ZTFLH:  TP393  
Corresponding Authors: Yan Dongmei，ORCID：0000-0002-5300-9493     E-mail: ydongmei@tjufe.edu.cn
Service
E-mail this article
Add to my bookshelf
Add to citation manager
E-mail Alert
RSS
Articles by authors
Dongmei Yan
Wenxin He
Zhi Chen

Cite this article:

Yan Dongmei, He Wenxin, Chen Zhi. Predicting Stock Prices Based on RoBERTa-TCN and Sentimental Characteristics. Data Analysis and Knowledge Discovery, 2022, 6(12): 123-134.

URL:

https://manu44.magtech.com.cn/Jwk_infotech_wk3/EN/10.11925/infotech.2096-3467.2022.0106     OR     https://manu44.magtech.com.cn/Jwk_infotech_wk3/EN/Y2022/V6/I12/123

Stock Price Prediction Model Based on RoBERTa-TCN
Schematic Diagram of Causal Convolution
Schematic Diagram of Dilated Convolution
Schematic Diagram of Residual Block
Schematic Diagram of Feature Fusion
参数 值
批处理大小（batch_size） 8
迭代次数（epoch） 30
TCN网络深度（depth） 3
TCN卷积核大小（kernel size） 3
优化器 AdamW
Parameter Settings
名称 所属板块 所属行业
平安银行000001.sz 深市 金融服务
五粮液 000858.sz 深市 白酒
上汽集团600104.sh 沪市 汽车
伊利股份600887.sh 沪市 食品饮料
乐普医疗300003.sz 创业板 医药生物
亿纬锂能300014.sz 创业板 锂电池
Stock Data Set Statistics
名称 数据量/条
平安银行000001.sz 98 444
五粮液 000858.sz 206 648
上汽集团600104.sh 125 404
伊利股份600887.sh 165 584
乐普医疗300003.sz 79 444
亿纬锂能300014.sz 94 421
Statistics of Stock Evaluation Data
模型 评价指标 平安银行 五粮液 上汽集团 伊利股份 乐普医疗 亿纬锂能
LSTM MAE 3.625 3 3.106 0 2.572 0 2.483 2 2.667 3 5.531 8
RMSE 4.597 3 4.163 7 3.521 1 3.404 5 3.386 5 7.216 5
MAPE 5.261 1 4.080 2 13.683 3 3.752 5 5.965 1 8.686 3
R2 0.906 3 0.668 8 0.908 6 0.875 4 0.845 7 0.752 4
GRU MAE 2.900 4 2.533 7 2.141 0 2.401 9 2.263 7 2.263 7
RMSE 3.745 2 3.308 1 3.037 3 3.244 0 2.894 9 2.894 9
MAPE 4.079 4 3.330 2 10.935 5 3.592 6 5.023 6 5.023 6
R2 0.946 7 0.837 4 0.930 6 0.897 1 0.883 9 0.883 9
TCN MAE 2.743 6 2.073 2 2.342 7 2.337 1 2.224 5 2.974 3
RMSE 3.593 9 2.679 0 3.190 9 3.123 5 2.854 7 3.946 9
MAPE 3.964 3 4.572 4 12.367 2 3.525 6 4.992 8 4.824 4
R2 0.948 9 0.902 3 0.928 4 0.904 8 0.886 8 0.952 4
RoBERTa-LSTM MAE 0.626 1 0.436 4 0.865 6 0.833 8 0.847 8 0.563 6
RMSE 0.896 4 0.666 1 1.184 7 1.105 4 1.192 5 1.046 7
MAPE 0.486 0 0.375 6 0.583 6 0.704 8 0.618 2 0.215 0
R2 0.992 0 0.995 6 0.986 0 0.987 7 0.985 7 0.989 1
RoBERTa-GRU MAE 0.651 6 0.498 4 0.860 9 0.794 2 0.835 6 0.723 4
RMSE 0.877 3 0.661 5 1.170 4 1.111 1 1.165 1 1.035 5
MAPE 0.659 2 0.530 3 0.487 7 0.694 8 0.649 7 0.185 0
R2 0.992 3 0.995 6 0.986 3 0.987 5 0.986 4 0.989 3
RoBERTa-TCN MAE 0.625 0 0.402 6 0.836 0 0.766 4 0.826 4 0.533 7
RMSE 0.867 2 0.609 0 1.155 2 1.047 3 1.153 6 0.973 5
MAPE 0.422 7 0.215 9 0.542 9 0.693 9 0.609 8 0.164 8
R2 0.992 5 0.996 3 0.986 7 0.988 9 0.986 6 0.991 2
Experimental Prediction Indexes of Each Model
模型 评价指标 平安银行 五粮液 上汽集团 伊利股份 乐普医疗 亿纬锂能
RoBERTa-TCN（5） MAE 0.625 0 0.402 6 0.836 0 0.766 4 0.826 4 0.533 7
RMSE 0.867 2 0.609 0 1.155 2 1.047 3 1.153 6 0.973 5
MAPE 0.422 7 0.215 9 0.542 9 0.693 9 0.609 8 0.164 8
R2 0.992 5 0.996 3 0.986 7 0.988 9 0.986 6 0.991 2
RoBERTa-TCN（10） MAE 0.541 3 0.378 3 0.829 6 0.714 5 0.854 5 0.559 7
RMSE 0.785 7 0.572 4 1.145 2 1.016 3 1.165 2 0.975 0
MAPE 0.510 8 0.244 7 0.512 3 0.614 0 0.639 7 0.229 8
R2 0.993 8 0.996 7 0.986 9 0.989 5 0.986 3 0.990 6
RoBERTa-TCN（15） MAE 0.544 4 0.435 6 0.871 2 0.762 8 0.851 6 0.575 3
RMSE 0.808 6 0.622 4 1.188 8 1.055 4 1.183 6 0.988 4
MAPE 0.429 6 0.470 0 0.566 8 0.661 1 0.657 7 0.174 5
R2 0.993 5 0.996 1 0.985 9 0.988 6 0.985 8 0.990 3
Experimental Predictors Using Sentimental Mean Values of Different Trading Days
模型 评价指标 平安银行 五粮液 上汽集团 伊利股份 乐普医疗 亿纬锂能
RoBERTa-TCN MAE 0.625 0 0.402 6 0.836 0 0.766 4 0.826 4 0.533 7
RMSE 0.867 2 0.609 0 1.155 2 1.047 3 1.153 6 0.973 5
MAPE 0.422 7 0.215 9 0.542 9 0.693 9 0.609 8 0.164 8
R2 0.992 5 0.996 3 0.986 7 0.988 9 0.986 6 0.991 2
RoBERTa-TCN-index MAE 0.560 1 0.314 0 0.816 3 0.726 5 0.862 9 0.511 0
RMSE 0.807 8 0.489 4 1.125 0 1.015 0 1.195 8 0.917 4
MAPE 0.462 5 0.224 7 0.512 9 0.736 6 0.780 4 0.176 9
R2 0.993 5 0.997 6 0.987 4 0.989 6 0.985 7 0.991 6
Experimental Prediction Indexes of Model with Stock Index
[1] Hochreiter S, Schmidhuber J. Long Short-Term Memory[J]. Neural Computation, 1997, 9(8): 1735-1780.
doi: 10.1162/neco.1997.9.8.1735 pmid: 9377276
[2] Cho K, van Merrienboer B, Gulcehre C, et al. Learning Phrase Representations Using RNN Encoder-Decoder for Statistical Machine Translation[C]// Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing. 2014: 1724-1734.
[3] Bai S J, Kolter J Z, Koltun V. An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling[OL]. arXiv Preprint, arXiv: 1803.01271.
[4] 赵澄, 叶耀威, 姚明海. 基于金融文本情感的股票波动预测[J]. 计算机科学, 2020, 47(5): 79-83.
doi: 10.11896/jsjkx.190400145
[4] (Zhao Cheng, Ye Yaowei, Yao Minghai. Stock Volatility Forecast Based on Financial Text Emotion[J]. Computer Science, 2020, 47(5): 79-83.)
doi: 10.11896/jsjkx.190400145
[5] Chen K, Zhou Y, Dai F Y. A LSTM-Based Method for Stock Returns Prediction: A Case Study of China Stock Market[C]// Proceedings of the 2015 IEEE International Conference on Big Data. 2015: 2823-2824.
[6] Fischer T, Krauss C. Deep Learning with Long Short-Term Memory Networks for Financial Market Predictions[J]. European Journal of Operational Research, 2018, 270(2): 654-669.
doi: 10.1016/j.ejor.2017.11.054
[7] 裴大卫, 朱明. 基于多因子与多变量长短期记忆网络的股票价格预测[J]. 计算机系统应用, 2019, 28(8): 30-38.
[7] (Pei Dawei, Zhu Ming. Stock Price Prediction Based on Multiple-Factor and Multi-Variable Long Short Term Memory[J]. Computer Systems & Applications, 2019, 28(8): 30-38.)
[8] Kim T, Kim H Y. Forecasting Stock Prices with a Feature Fusion LSTM-CNN Model Using Different Representations of the Same Data[J]. PLoS One, 2019, 14(2): e0212320.
[9] 张倩玉, 严冬梅, 韩佳彤. 结合深度学习和分解算法的股票价格预测研究[J]. 计算机工程与应用, 2021, 57(5): 56-64.
doi: 10.3778/j.issn.1002-8331.2006-0444
[9] (Zhang Qianyu, Yan Dongmei, Han Jiatong. Research on Stock Price Prediction Combined with Deep Learning and Decomposition Algorithm[J]. Computer Engineering and Applications, 2021, 57(5): 56-64.)
doi: 10.3778/j.issn.1002-8331.2006-0444
[10] 陈禹帆, 温蜜, 张凯, 等. 基于相似日匹配及TCN-Attention的短期光伏出力预测[J]. 电测与仪表, 2022, 59(10): 108-116.
[10] (Chen Yufan, Wen Mi, Zhang Kai, et al. Short-Term Photovoltaic Output Forecasting Based on Similar Day Matching and TCN-Attention[J]. Electrical Measurement & Instrumentation, 2022, 59(10): 108-116.)
[11] 杨娟. 互联网财经新闻对股票影响的实证分析: 基于公司新闻语义分析的视角[D]. 成都: 西南财经大学, 2012.
[11] (Yang Juan. Empirical Analysis of the Impact of Internet Financial News on Stock: Analysis of Company News[D]. Chengdu: Southwestern University of Finance and Economics, 2012.)
[12] 赵亚南, 刘渊, 宋设. 融合多头自注意力机制的金融新闻极性分析[J]. 计算机工程, 2020, 46(8): 85-92.
[12] (Zhao Yanan, Liu Yuan, Song She. Financial News Polarity Analysis Fusing with Multi-Head Self-Attention Mechanism[J]. Computer Engineering, 2020, 46(8): 85-92.)
[13] 王婷, 杨文忠. 文本情感分析方法研究综述[J]. 计算机工程与应用, 2021, 57(12): 11-24.
doi: 10.3778/j.issn.1002-8331.2101-0022
[13] (Wang Ting, Yang Wenzhong. Review of Text Sentiment Analysis Methods[J]. Computer Engineering and Applications, 2021, 57(12): 11-24.)
doi: 10.3778/j.issn.1002-8331.2101-0022
[14] Ren R, Wu D D, Liu T X. Forecasting Stock Market Movement Direction Using Sentiment Analysis and Support Vector Machine[J]. IEEE Systems Journal, 2019, 13(1): 760-770.
doi: 10.1109/JSYST.2018.2794462
[15] Dong J H. Financial Investor Sentiment Analysis Based on FPGA and Convolutional Neural Network[J]. Microprocessors and Microsystems, 2020: 103418.
[16] Devlin J, Chang M W, Lee Kenton, et al. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding[OL]. arXiv Preprint, arXiv: 1810.04805.
[17] Munikar M, Shakya S, Shrestha A. Fine-Grained Sentiment Classification Using BERT[C]// Proceedings of 2019 Artificial Intelligence for Transforming Business and Society. 2019: 1-5.
[18] Liu Y H, Ott M, Goyal N, et al. BoBERTa: A Robustly Optimized BERT Pretraining Approach[OL]. arXiv Preprint, arXiv: 1907. 11692.
[19] Kaminski J. Nowcasting the Bitcoin Market with Twitter Signals[OL]. arXiv Preprint, arXiv: 1406.7577.
[20] 樊鹏英, 杨音, 张正平, 等. 个股投资者情绪与股票收益率的关系——基于股评信息视角的研究[J]. 数学的实践与认识, 2021, 51(16): 305-320.
[20] (Fan Pengying, Yang Yin, Zhang Zhengping, et al. The Relationship Between Individual Stock Investor Sentiment and the Stock Yield——Based on the Perspective of Stock Evaluation Information[J]. Mathematics in Practice and Theory, 2021, 51(16): 305-320.)
[21] 蔡启航. 媒体情绪对股票收益率的影响分析——基于自然语言处理和多因子模型的拓展研究[D]. 杭州: 浙江大学, 2021.
[21] (Cai Qihang. Analysis of the Influence of Media Sentiment on Stock Returns——An Extended Research Based on Natural Language Processing and Multi-Factor Models[D]. Hangzhou: Zhejiang University, 2021.)
[22] Jing N, Wu Z, Wang H F. A Hybrid Model Integrating Deep Learning with Investor Sentiment Analysis for Stock Price Prediction[J]. Expert Systems with Applications, 2021, 178: 115019.
doi: 10.1016/j.eswa.2021.115019
[23] Vargas M R, dos Anjos C E M, Bichara G L G, et al. Deep Learning for Stock Market Prediction Using Technical Indicators and Financial News Articles[C]// Proceedings of 2018 International Joint Conference on Neural Networks. 2018: 1-8.
[24] 徐月梅, 王子厚, 吴子歆. 一种基于CNN-BiLSTM多特征融合的股票走势预测模型[J]. 数据分析与知识发现, 2021, 5(7): 126-137.
[24] (Xu Yuemei, Wang Zihou, Wu Zixin. Predicting Stock Trends with CNN-BiLSTM Based Multi-Feature Integration Model[J]. Data Analysis and Knowledge Discovery, 2021, 5(7): 126-137.)
[25] Li X D, Wu P J, Wang W P. Incorporating Stock Prices and News Sentiments for Stock Market Prediction: A Case of Hong Kong[J]. Information Processing & Management, 2020, 57(5): 102212.
doi: 10.1016/j.ipm.2020.102212
[26] Kanavos A, Vonitsanos G, Mohasseb A, et al. An Entropy-Based Evaluation for Sentiment Analysis of Stock Market Prices Using Twitter Data[C]// Proceedings of 2020 15th International Workshop on Semantic and Social Media Adaptation and Personalization. 2020: 1-7.
[27] Ruder S, Peters M E, Swayamdipta S, et al. Transfer Learning in Natural Language Processing[C]// Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics:Tutorials. 2019: 15-18.
[28] Araci D. FinBERT: Financial Sentiment Analysis with Pre-trained Language Models[OL]. arXiv Preprint, arXiv: 1908.10063.
[29] Li M G, Li W R, Wang F, et al. Applying BERT to Analyze Investor Sentiment in Stock Market[J]. Neural Computing and Applications, 2021, 33(10): 4663-4676.
doi: 10.1007/s00521-020-05411-7
[30] Li M Z, Chen L, Zhao J, et al. Sentiment Analysis of Chinese Stock Reviews Based on BERT Model[J]. Applied Intelligence, 2021, 51(7): 5016-5024.
doi: 10.1007/s10489-020-02101-8
[31] van den Oord A, Dieleman S, Zen H G, et al. WaveNet: A Generative Model for Raw Audio[OL]. arXiv Preprint, arXiv: 1609.03499.
[32] He K M, Zhang X Y, Ren S Q, et al. Deep Residual Learning for Image Recognition[C]// Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. 2016: 770-778.
[33] 杨浩东. 基于深度注意力机制的视频中人体动作识别[D]. 长沙: 国防科技大学, 2018.
[33] (Yang Haodong. Attention Mechanism Based Deep Network for Human Action Recognition in Video[D]. Changsha: National University of Defense Technology, 2018.)
[34] Vaswani A, Shazeer N, Parmar N, et al. Attention is All You Need[C]// Proceedings of the 31st International Conference on Neural Information Processing Systems. 2017: 6000-6010.
[35] Jin Z G, Yang Y, Liu Y H. Stock Closing Price Prediction Based on Sentiment Analysis and LSTM[J]. Neural Computing and Applications, 2020, 32(13): 9713-9729.
doi: 10.1007/s00521-019-04504-2
[1] Zhao Ruijie, Tong Xinyu, Liu Xiaohua, Lu Yonghe. Entity Recognition and Labeling for Medical Literature Based on Neural Network[J]. 数据分析与知识发现, 2022, 6(9): 100-112.
[2] Chen Yuanyuan, Ma Jing. Detecting Multimodal Sarcasm Based on SC-Attention Mechanism[J]. 数据分析与知识发现, 2022, 6(9): 40-51.
[3] Tang Jiao, Zhang Lisheng, Sang Chunyan. News Recommendation with Latent Topic Distribution and Long and Short-Term User Representations[J]. 数据分析与知识发现, 2022, 6(9): 52-64.
[4] Zhao Pengwu, Li Zhiyi, Lin Xiaoqi. Identifying Relationship of Chinese Characters with Attention Mechanism and Convolutional Neural Network[J]. 数据分析与知识发现, 2022, 6(8): 41-51.
[5] Ye Han,Sun Haichun,Li Xin,Jiao Kainan. Classification Model for Long Texts with Attention Mechanism and Sentence Vector Compression[J]. 数据分析与知识发现, 2022, 6(6): 84-94.
[6] Zhang Ruoqi, Shen Jianfang, Chen Pinghua. Session Sequence Recommendation with GNN, Bi-GRU and Attention Mechanism[J]. 数据分析与知识发现, 2022, 6(6): 46-54.
[7] Guo Hangcheng, He Yanqing, Lan Tian, Wu Zhenfeng, Dong Cheng. Identifying Moves from Scientific Abstracts Based on Paragraph-BERT-CRF[J]. 数据分析与知识发现, 2022, 6(2/3): 298-307.
[8] Xu Yuemei, Fan Zuwei, Cao Han. A Multi-Task Text Classification Model Based on Label Embedding of Attention Mechanism[J]. 数据分析与知识发现, 2022, 6(2/3): 105-116.
[9] Zhang Yunqiu, Wang Yang, Li Bocheng. Identifying Named Entities of Chinese Electronic Medical Records Based on RoBERTa-wwm Dynamic Fusion Model[J]. 数据分析与知识发现, 2022, 6(2/3): 242-250.
[10] Fan Tao, Wang Hao, Zhang Wei, Li Xiaomin. Extracting Entities from Intangible Cultural Heritage Texts Based on Machine Reading Comprehension[J]. 数据分析与知识发现, 2022, 6(12): 70-79.
[11] Bai Simeng,Niu Zhendong,He Hui,Shi Kaize,Yi Kun,Ma Yuanchi. Biomedical Text Classification Method Based on Hypergraph Attention Network[J]. 数据分析与知识发现, 2022, 6(11): 13-24.
[12] Huang Xuejian, Liu Yuyang, Ma Tinghuai. Classification Model for Scholarly Articles Based on Improved Graph Neural Network[J]. 数据分析与知识发现, 2022, 6(10): 93-102.
[13] Yu Xuehan, He Lin, Xu Jian. Extracting Events from Ancient Books Based on RoBERTa-CRF[J]. 数据分析与知识发现, 2021, 5(7): 26-35.
[14] Yang Hanxun, Zhou Dequn, Ma Jing, Luo Yongcong. Detecting Rumors with Uncertain Loss and Task-level Attention Mechanism[J]. 数据分析与知识发现, 2021, 5(7): 101-110.
[15] Xie Hao,Mao Jin,Li Gang. Sentiment Classification of Image-Text Information with Multi-Layer Semantic Fusion[J]. 数据分析与知识发现, 2021, 5(6): 103-114.
  Copyright © 2016 Data Analysis and Knowledge Discovery   Tel/Fax:(010)82626611-6626，82624938   E-mail:jishu@mail.las.ac.cn