|
|
|
| Text Sentiment Classification Algorithm Based on Prompt Learning Enhancement |
Huang Taifeng,Ma Jing( ) |
| College of Economics and Management, Nanjing University of Aeronautics and Astronautics, Nanjing 211106, China |
|
|
|
|
Abstract [Objective] This paper aims to improve the low accuracy of sentiment classification using the pre-trained model with insufficient samples.[Methods] We proposed a prompt learning enhanced sentiment classification algorithm Pe(prompt ensemble)-RoBERTa. It modified the RoBERTa model with integrated prompts different from the traditional fine-tuning methods. The new model could understand the downstream tasks and extract the text’s sentiment features. [Results] We examined the model on several publicly accessible Chinese and English datasets. The average sentiment classification accuracy of the model reached 93.2% with fewer samples. Compared with fine-tuned and discrete prompts, our new model’s accuracy improved by 13.8% and 8.1%, respectively. [Limitations] The proposed model only processes texts for the sentiment dichotomization tasks. It did not involve the more fine-grained sentiment classification tasks. [Conclusions] The Pe-RoBERTa model can extract text sentiment features and achieve high accuracy in sentiment classification tasks.
|
|
Received: 03 January 2023
Published: 08 January 2024
|
|
|
| Fund:National Natural Science Foundation of China(72174086);Nanjing University of Aeronautics and Astronautics Graduate Researchand Practice Innovation Project(xcxjh20220910) |
|
Corresponding Authors:
Ma Jing，ORCID：0000-0001-8472-2518，E-mail：majing5525@126.com。
|
| [1] |
洪巍, 李敏. 文本情感分析方法研究综述[J]. 计算机工程与科学, 2019, 41(4): 750-757.
|
| [1] |
(Hong Wei, Li Min. A Review: Text Sentiment Analysis Methods[J]. Computer Engineering & Science, 2019, 41(4): 750-757.)
|
| [2] |
曾慧玲, 李琳, 吕思洋, 等. 提示学习驱动的新闻舆情风险识别方法研究[J]. 计算机工程与应用, 2024, 60(1): 182-188.
doi: 10.3778/j.issn.1002-8331.2208-0295
|
| [2] |
(Zeng Huiling, Li Lin, Lyu Siyang, et al. Risk Identification Method for News Public Opinion Driven by Prompt Learning[J]. Computer Engineering and Applications, 2024, 60(1): 182-188.)
doi: 10.3778/j.issn.1002-8331.2208-0295
|
| [3] |
万家山, 吴云志. 基于深度学习的文本分类方法研究综述[J]. 天津理工大学学报, 2021, 37(2): 41-47.
|
| [3] |
(Wan Jiashan, Wu Yunzhi. Review of Text Classification Research Based on Deep Learning[J]. Journal of Tianjin University of Technology, 2021, 37(2): 41-47.)
|
| [4] |
李杰, 李欢. 基于深度学习的短文本评论产品特征提取及情感分类研究[J]. 情报理论与实践, 2018, 41(2): 143-148.
|
| [4] |
(Li Jie, Li Huan. Research on Product Feature Extraction and Sentiment Classification of Short Online Review Based on Deep Learning[J]. Information Studies: Theory & Application, 2018, 41(2): 143-148.)
|
| [5] |
Turian J, Ratinov L A, Bengio Y. Word Representations: A Simple and General Method for Semi-supervised Learning[C]// Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics. 2010: 384-394.
|
| [6] |
Mikolov T, Sutskever I, Chen K, et al. Distributed Representations of Words and Phrases and Their Compositionality[OL]. arXiv Preprint, arXiv: 1310.4546.
|
| [7] |
Pennington J, Socher R, Manning C. GloVe: Global Vectors for Word Representation[C]// Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing. 2014: 1532-1543.
|
| [8] |
Bojanowski P, Grave E, Joulin A, et al. Enriching Word Vectors with Subword Information[J]. Transactions of the Association for Computational Linguistics, 2017, 5: 135-146.
doi: 10.1162/tacl_a_00051
|
| [9] |
Devlin J, Chang M W, Lee K, et al. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding[C]// Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics:Human Language Technologies, Volume 1 (Long and Short Papers). 2019: 4171-4186.
|
| [10] |
Liu Y H, Ott M, Goyal N, et al. RoBERTa: A Robustly Optimized BERT Pretraining Approach[OL]. arXiv Preprint, arXiv:1907.11692.
|
| [11] |
Lewis M, Liu Y H, Goyal N, et al. BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension[C]//Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 2020: 7871-7880.
|
| [12] |
Dong L, Yang N, Wang W H, et al. Unified Language Model Pre-training for Natural Language Understanding and Generation[C]// Proceedings of the 33rd International Conference on Neural Information Processing Systems. 2019: 13063-13075.
|
| [13] |
Vaswani A, Shazeer N, Parmar N, et al. Attention is All You Need[OL]. arXiv Preprint, arXiv:1706.03762.
|
| [14] |
Howard J, Ruder S. Universal Language Model Fine-Tuning for Text Classification[C]// Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1:Long Papers). 2018: 328-339.
|
| [15] |
Brown T B, Mann B, Ryder N, et al. Language Models are Few-Shot Learners[OL]. arXiv Preprint, arXiv: 2005.14165.
|
| [16] |
赵凯琳, 靳小龙, 王元卓. 小样本学习研究综述[J]. 软件学报, 2021, 32(2): 349-369.
|
| [16] |
(Zhao Kailin, Jin Xiaolong, Wang Yuanzhuo. Survey on Few-Shot Learning[J]. Journal of Software, 2021, 32(2): 349-369.)
|
| [17] |
Li X L, Liang P. Prefix-Tuning: Optimizing Continuous Prompts for Generation[OL]. arXiv Preprint, arXiv: 2101.00190.
|
| [18] |
Cui L Y, Wu Y, Liu J, et al. Template-Based Named Entity Recognition Using BART[C]// Findings of the Association for Computational Linguistics:ACL-IJCNLP 2021. 2021: 1835-1845.
|
| [19] |
Petroni F, Rocktäschel T, Riedel S, et al. Language Models as Knowledge Bases?[C]// Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing. 2019: 2463-2473.
|
| [20] |
Shin T, Razeghi Y, Logan IV R L, et al. AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts[C]// Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing. 2020: 4222-4235.
|
| [21] |
Hu S D, Ding N, Wang H D, et al. Knowledgeable Prompt-Tuning: Incorporating Knowledge into Prompt Verbalizer for Text Classification[C]// Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1:Long Papers). 2022: 2225-2240.
|
| [22] |
Gao T Y, Fisch A, Chen D Q. Making Pre-trained Language Models Better Few-Shot Learners[OL]. arXiv Preprint, arXiv: 2012.15723.
|
| [23] |
Wallace E, Feng S, Kandpal N, et al. Universal Adversarial Triggers for Attacking and Analyzing NLP[OL]. arXiv Preprint, arXiv: 1908.07125.
|
| [24] |
Yuan W Z, Neubig G, Liu P F. BARTScore: Evaluating Generated Text as Text Generation[OL]. arXiv Preprint, arXiv: 2106.11520.
|
| [25] |
Raffel C, Shazeer N, Roberts A, et al. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer[OL]. arXiv Preprint, arXiv: 1910.10683.
|
| [26] |
李南星. 基于BERT和提示学习的改进句向量文本表示[D]. 汕头: 汕头大学, 2022.
|
| [26] |
(Li Nanxing. Improved Sentence Embedding Based on BERT and Prompt-Learning[D]. Shantou: Shantou University, 2022.)
|
| [27] |
Liu X, Zheng Y N, Du Z X, et al. GPT Understands, Too[OL]. arXiv Preprint, arXiv: 2103.10385.
|
| [28] |
Zhong Z X, Friedman D, Chen D Q. Factual Probing is [MASK]: Learning vs. Learning to Recall[C]//Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2021: 5017-5033.
|
| [29] |
Lester B, Al-Rfou R, Constant N. The Power of Scale for Parameter-Efficient Prompt Tuning[OL]. arXiv Preprint, arXiv: 2104.08691.
|
| [30] |
Paszke A, Gross S, Massa F, et al. PyTorch: An Imperative Style, High-Performance Deep Learning Library[C]// Proceedings of the 33rd International Conference on Neural Information Processing Systems. 2019: 8026-8037.
|
| [31] |
Wolf T, Debut L, Sanh V, et al. HuggingFace’s Transformers: State-of-the-Art Natural Language Processing[OL]. arXiv Preprint, arXiv: 1910.03771.
|
| [32] |
周志华. 机器学习[M]. 北京: 清华大学出版社, 2016.
|
| [32] |
(Zhou Zhihua. Machine Learning[M]. Beijing: Tsinghua University Press, 2016.)
|
|
Viewed |
|
|
|
Full text
|
|
|
|
|
Abstract
|
|
|
|
|
Cited |
|
|
|
|
| |
Shared |
|
|
|
|
| |
Discussed |
|
|
|
|
