[Objective] This paper aims to improve the low accuracy of sentiment classification using the pre-trained model with insufficient samples.[Methods] We proposed a prompt learning enhanced sentiment classification algorithm Pe(prompt ensemble)-RoBERTa. It modified the RoBERTa model with integrated prompts different from the traditional fine-tuning methods. The new model could understand the downstream tasks and extract the text’s sentiment features. [Results] We examined the model on several publicly accessible Chinese and English datasets. The average sentiment classification accuracy of the model reached 93.2% with fewer samples. Compared with fine-tuned and discrete prompts, our new model’s accuracy improved by 13.8% and 8.1%, respectively. [Limitations] The proposed model only processes texts for the sentiment dichotomization tasks. It did not involve the more fine-grained sentiment classification tasks. [Conclusions] The Pe-RoBERTa model can extract text sentiment features and achieve high accuracy in sentiment classification tasks.
黄泰峰, 马静. 基于提示学习增强的文本情感分类模型*[J]. 数据分析与知识发现, 2024, 8(3): 77-84.
Huang Taifeng, Ma Jing. Text Sentiment Classification Algorithm Based on Prompt Learning Enhancement. Data Analysis and Knowledge Discovery, 2024, 8(3): 77-84.
(Zeng Huiling, Li Lin, Lyu Siyang, et al. Risk Identification Method for News Public Opinion Driven by Prompt Learning[J]. Computer Engineering and Applications, 2024, 60(1): 182-188.)
doi: 10.3778/j.issn.1002-8331.2208-0295
(Wan Jiashan, Wu Yunzhi. Review of Text Classification Research Based on Deep Learning[J]. Journal of Tianjin University of Technology, 2021, 37(2): 41-47.)
(Li Jie, Li Huan. Research on Product Feature Extraction and Sentiment Classification of Short Online Review Based on Deep Learning[J]. Information Studies: Theory & Application, 2018, 41(2): 143-148.)
[5]
Turian J, Ratinov L A, Bengio Y. Word Representations: A Simple and General Method for Semi-supervised Learning[C]// Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics. 2010: 384-394.
[6]
Mikolov T, Sutskever I, Chen K, et al. Distributed Representations of Words and Phrases and Their Compositionality[OL]. arXiv Preprint, arXiv: 1310.4546.
[7]
Pennington J, Socher R, Manning C. GloVe: Global Vectors for Word Representation[C]// Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing. 2014: 1532-1543.
[8]
Bojanowski P, Grave E, Joulin A, et al. Enriching Word Vectors with Subword Information[J]. Transactions of the Association for Computational Linguistics, 2017, 5: 135-146.
doi: 10.1162/tacl_a_00051
[9]
Devlin J, Chang M W, Lee K, et al. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding[C]// Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics:Human Language Technologies, Volume 1 (Long and Short Papers). 2019: 4171-4186.
[10]
Liu Y H, Ott M, Goyal N, et al. RoBERTa: A Robustly Optimized BERT Pretraining Approach[OL]. arXiv Preprint, arXiv:1907.11692.
[11]
Lewis M, Liu Y H, Goyal N, et al. BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension[C]//Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 2020: 7871-7880.
[12]
Dong L, Yang N, Wang W H, et al. Unified Language Model Pre-training for Natural Language Understanding and Generation[C]// Proceedings of the 33rd International Conference on Neural Information Processing Systems. 2019: 13063-13075.
[13]
Vaswani A, Shazeer N, Parmar N, et al. Attention is All You Need[OL]. arXiv Preprint, arXiv:1706.03762.
[14]
Howard J, Ruder S. Universal Language Model Fine-Tuning for Text Classification[C]// Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1:Long Papers). 2018: 328-339.
[15]
Brown T B, Mann B, Ryder N, et al. Language Models are Few-Shot Learners[OL]. arXiv Preprint, arXiv: 2005.14165.
(Zhao Kailin, Jin Xiaolong, Wang Yuanzhuo. Survey on Few-Shot Learning[J]. Journal of Software, 2021, 32(2): 349-369.)
[17]
Li X L, Liang P. Prefix-Tuning: Optimizing Continuous Prompts for Generation[OL]. arXiv Preprint, arXiv: 2101.00190.
[18]
Cui L Y, Wu Y, Liu J, et al. Template-Based Named Entity Recognition Using BART[C]// Findings of the Association for Computational Linguistics:ACL-IJCNLP 2021. 2021: 1835-1845.
[19]
Petroni F, Rocktäschel T, Riedel S, et al. Language Models as Knowledge Bases?[C]// Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing. 2019: 2463-2473.
[20]
Shin T, Razeghi Y, Logan IV R L, et al. AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts[C]// Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing. 2020: 4222-4235.
[21]
Hu S D, Ding N, Wang H D, et al. Knowledgeable Prompt-Tuning: Incorporating Knowledge into Prompt Verbalizer for Text Classification[C]// Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1:Long Papers). 2022: 2225-2240.
[22]
Gao T Y, Fisch A, Chen D Q. Making Pre-trained Language Models Better Few-Shot Learners[OL]. arXiv Preprint, arXiv: 2012.15723.
[23]
Wallace E, Feng S, Kandpal N, et al. Universal Adversarial Triggers for Attacking and Analyzing NLP[OL]. arXiv Preprint, arXiv: 1908.07125.
[24]
Yuan W Z, Neubig G, Liu P F. BARTScore: Evaluating Generated Text as Text Generation[OL]. arXiv Preprint, arXiv: 2106.11520.
[25]
Raffel C, Shazeer N, Roberts A, et al. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer[OL]. arXiv Preprint, arXiv: 1910.10683.
[26]
李南星. 基于BERT和提示学习的改进句向量文本表示[D]. 汕头: 汕头大学, 2022.
[26]
(Li Nanxing. Improved Sentence Embedding Based on BERT and Prompt-Learning[D]. Shantou: Shantou University, 2022.)
[27]
Liu X, Zheng Y N, Du Z X, et al. GPT Understands, Too[OL]. arXiv Preprint, arXiv: 2103.10385.
[28]
Zhong Z X, Friedman D, Chen D Q. Factual Probing is [MASK]: Learning vs. Learning to Recall[C]//Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2021: 5017-5033.
[29]
Lester B, Al-Rfou R, Constant N. The Power of Scale for Parameter-Efficient Prompt Tuning[OL]. arXiv Preprint, arXiv: 2104.08691.
[30]
Paszke A, Gross S, Massa F, et al. PyTorch: An Imperative Style, High-Performance Deep Learning Library[C]// Proceedings of the 33rd International Conference on Neural Information Processing Systems. 2019: 8026-8037.
[31]
Wolf T, Debut L, Sanh V, et al. HuggingFace’s Transformers: State-of-the-Art Natural Language Processing[OL]. arXiv Preprint, arXiv: 1910.03771.
[32]
周志华. 机器学习[M]. 北京: 清华大学出版社, 2016.
[32]
(Zhou Zhihua. Machine Learning[M]. Beijing: Tsinghua University Press, 2016.)
