ChatGPT中文性能测评与风险应对<sup>*</sup>

doi:10.11925/infotech.2096-3467.2023.0214

数据分析与知识发现

2023, Vol. 7

Issue (3): 16-25 https://doi.org/10.11925/infotech.2096-3467.2023.0214

专题

本期目录 | 过刊浏览 | 高级检索

ChatGPT中文性能测评与风险应对^*

张华平(

),李林翰,李春锦

北京理工大学计算机学院北京 100081

ChatGPT Performance Evaluation on Chinese Language and Risk Measures

Zhang Huaping(

),Li Linhan,Li Chunjin

School of Computer Science, Beijing Institute of Technology, Beijing 100081, China

摘要
图/表
参考文献
相关文章
Metrics

全文: PDF (798 KB) HTML ( 75 )
输出: BibTeX | EndNote (RIS)

摘要

【目的】 简要介绍ChatGPT的主要技术创新，使用9个数据集在4个任务上测评ChatGPT的中文性能，分析ChatGPT潜在的风险以及中国应对策略建议。【方法】 使用ChnSentiCorp数据集测试ChatGPT和WeLM模型，EPRSTMT数据集测试ChatGPT和ERNIE 3.0 Titan，发现ChatGPT在情感分析任务上与国产大模型差距不大；使用LCSTS和TTNews数据集测试ChatGPT和WeLM模型，ChatGPT均优于WeLM；使用CMRC2018和DRCD数据集进行抽取式的机器阅读理解，C³数据集进行常识性的机器阅读理解，发现ERNIE 3.0 Titan在该任务中优于ChatGPT；使用WebQA和CKBQA数据集进行中文闭卷问答测试，发现ChatGPT容易产生事实性错误，与国产模型差距较大。【结果】 ChatGPT在自然语言处理的经典任务上表现较好，在情感分析上具有85%以上的准确率，在闭卷问答上出现事实性错误的概率较高。【局限】 将判别式的任务转为生成式的过程中可能引入评估分数的误差。本文仅在零样本情况下评估ChatGPT，并不清楚其在其他情况下的表现。由于后续版本的迭代更新，ChatGPT评测结果具有一定的时效性。【结论】 ChatGPT很强大但仍然存在一些缺点，研发中国化的大模型应以国家战略需求为导向，并且注意语言模型的风险和局限。

	服务

	把本文推荐给朋友
	加入引用管理器
	E-mail Alert
	RSS
	作者相关文章
	张华平
	李林翰
	李春锦

关键词 ：语言模型, ChatGPT, 人工智能

Abstract：

[Objective] This paper briefly introduces the main technical innovations of ChatGPT, and evaluates the performance of ChatGPT in Chinese on four tasks using nine datasets, analyzes the risk with ChatGPT and proposes our solutions. [Methods] ChatGPT and WeLM models were tested using the ChnSentiCorp dataset, and ChatGPT and ERNIE 3.0 Titan were tested using the EPRSTMT dataset, and it was found that ChatGPT did not differ much from the large domestic models in sentiment analysis tasks. The LCSTS and TTNews datasets were used to test the ChatGPT and WeLM models, and both ChatGPT outperformed the WeLM model; CMRC2018 and DRCD were used for extractive machine reading comprehension(MRC), and the C³ dataset was used for common sense MRC, and it was found that ERNIE 3.0 Titan outperformed ChatGPT in this task. WebQA and CKBQA were used to do Chinese closed-book quiz testing, and it was found that ChatGPT was prone to make factual errors in this task, and the domestic model outperformed ChatGPT. [Results] ChatGPT performed well on classic tasks of natural language processing, such as sentiment analysis with an accuracy rate of more than 85% and a higher probability of factual errors on closed-book questions. [Limitations] The error of evaluation score may be introduced in the process of converting discriminative tasks into generative ones. This paper only evaluated ChatGPT in zero-shot case, so it is not clear how it performs in other cases. ChatGPT may be updated iteratively in subsequent releases, and the profiling results may be time-sensitive. [Conclusions] ChatGPT is powerful but still has some drawbacks, for the large model of Chinese need to be national strategy oriented and pay attention to the limitations of the language model.

Key words： Language Model ChatGPT Artificial Intelligence

收稿日期: 2023-03-13 出版日期: 2023-03-16

ZTFLH:

TP391

基金资助:北京市自然科学基金(4212026);基础加强计划技术领域基金(2021-JCJQ-JJ-0059)

通讯作者: 张华平，ORCID:0000-0002-0137-4069，E-mail:kevinzhang@bit.edu.cn。

引用本文:

张华平, 李林翰, 李春锦. ChatGPT中文性能测评与风险应对^*[J]. 数据分析与知识发现, 2023, 7(3): 16-25.
Zhang Huaping, Li Linhan, Li Chunjin. ChatGPT Performance Evaluation on Chinese Language and Risk Measures. Data Analysis and Knowledge Discovery, 2023, 7(3): 16-25.

链接本文:

https://manu44.magtech.com.cn/Jwk_infotech_wk3/CN/10.11925/infotech.2096-3467.2023.0214 或 https://manu44.magtech.com.cn/Jwk_infotech_wk3/CN/Y2023/V7/I3/16

Fig.1 生成式语言模型的解码过程

Fig.2 ChatGPT的开发过程

Table 1 中文情感分析任务的性能测试实验结果

Table 2 情感分析任务的输入输出示例

Table 3 中文文本摘要任务的性能测试实验

Table 4 文本摘要任务的输入输出示例

Table 5 中文机器阅读理解任务的性能测试实验

Table 6 机器阅读理解任务的输入输出示例

Table 7 中文闭卷问答的性能测试实验

Table 8 中文闭卷问答任务的输入输出示例

Fig.3 ChatGPT混淆人物典故的示例

Table 9 ChatGPT编造的简历与可能的信息来源

[1]	Brown T, Mann B, Ryder N, et al. Language Models are Few-Shot Learners[C]// Proceedings of the 34th International Conference on Neural Information Processing Systems. 2020, 33: 1877-1901.
[2]	Thoppilan R, De Freitas D, Hall J, et al. LaMDA: Language Models for Dialog Applications [OL]. arXiv Preprint, arXiv:2201.08239.
[3]	Wang S H, Sun Y, Xiang Y, et al. ERNIE 3.0 Titan: Exploring Larger-scale Knowledge Enhanced Pre-training for Language Understanding and Generation[OL]. arXiv Preprint, arXiv:2112.12731.
[4]	Zeng W, Ren X, Su T, et al. PanGu-α: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-Parallel Computation[OL]. arXiv Preprint, arXiv:2104.12369.
[5]	Su H, Zhou X, Yu H J, et al. WeLM: A Well-Read Pre-trained Language Model for Chinese[OL]. arXiv Preprint, arXiv:2209.10372.
[6]	Kiela D, Bartolo M, Nie Y X, et al. Dynabench: Rethinking Benchmarking in NLP[C]// Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics:Human Language Technologies. 2021: 4110-4124.
[7]	Zhou J, Ke P, Qiu X P, et al. ChatGPT: Potential, Prospects, and Limitations[J]. Frontiers of Information Technology & Electronic Engineering. https://doi.org/10.1631/FITEE.2300089.
[8]	van Dis E, Bollen J, Zuidema W, et al. ChatGPT: Five Priorities for Research[J]. Nature, 2023, 614(7947): 224-226. doi: 10.1038/d41586-023-00288-7
[9]	Thorp H H. ChatGPT is Fun, but Not an Author[J]. Science, 2023, 379(6630): 313. doi: 10.1126/science.adg7879 pmid: 36701446
[10]	Qin C W, Zhang A, Zhang Z S, et al. Is ChatGPT a General-Purpose Natural Language Processing Task Solver? [OL]. arXiv Preprint, arXiv:2302.06476.
[11]	Bang Y, Cahyawijaya S, Lee N, et al. A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity[OL]. arXiv Preprint, arXiv:2302.04023.
[12]	Chen X T, Ye J J, Zu C, et al. How Robust is GPT-3.5 to Predecessors? A Comprehensive Study on Language Understanding Tasks[OL]. arXiv Preprint, arXiv:2303.00293.
[13]	Jiao W X, Wang W X, Huang J T, et al. Is ChatGPT a Good Translator? A Preliminary Study[OL]. arXiv Preprint, arXiv:2301.08745.
[14]	Devlin J, Chang M W, Lee K, et al. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding[C]// Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics:Human Language Technologies, Volume 1 (Long and Short Papers). 2019: 4171-4186.
[15]	Radford A, Narasimhan K, Salimans T, et al. Improving Language Understanding by Generative Pre-training[OL]. https://gwern.net/doc/www/s3-us-west-2.amazonaws.com/d73fdc5ffa8627bce44dcda2fc012da638ffb158.pdf.
[16]	Radford A, Wu J, Child R, et al. Language Models are Unsupervised Multitask Learners[OL]. OpenAI Blog. .https://gwern.net/doc/ai/nn/transformer/gpt/2019-radford.pdf
[17]	Elman J L. Finding Structure in Time[J]. Cognitive Science, 1990, 14(2): 179-211. doi: 10.1207/s15516709cog1402_1
[18]	Hochreiter S, Schmidhuber J. Long Short-Term Memory[J]. Neural Computation, 1997, 9(8): 1735-1780. doi: 10.1162/neco.1997.9.8.1735 pmid: 9377276
[19]	Vaswani A, Shazeer N, Parmar N, et al. Attention is All You Need[C]// Proceedings of the 31st International Conference on Neural Information Processing Systems. 2017: 6000-6010.
[20]	Chen M, Tworek J, Jun H, et al. Evaluating Large Language Models Trained on Code[OL]. arXiv Preprint, arXiv:2107.03374.
[21]	Wei J, Bosma M, Zhao V Y, et al. Finetuned Language Models are Zero-Shot Learners[OL]. arXiv Preprint, arXiv:2109.01652.
[22]	Zhang Y Z, Sun S Q, Galley M, et al. DialoGPT: Large-scale Generative Pre-training for Conversational Response Generation[C]// Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics:System Demonstrations. 2020: 270-278.
[23]	Nakano R, Hilton J, Balaji S, et al. WebGPT: Browser-assisted Question-Answering with Human Feedback[OL]. arXiv Preprint, arXiv:2112.09332.
[24]	Ouyang L, Wu J, Jiang X, et al. Training Language Models to Follow Instructions with Human Feedback[OL]. arXiv Preprint, arXiv:2203.02155.
[25]	Christiano P F, Leike J, Brown T, et al. Deep Reinforcement Learning from Human Preferences[C]// Proceedings of the 31st International Conference on Neural Information Processing Systems. 2017: 4302-4310.
[26]	Schulman J, Wolski F, Dhariwal P, et al. Proximal Policy Optimization Algorithms[OL]. arXiv Preprint, arXiv:1707.06347.
[27]	Xu L, Lu X, Yuan C, et al. FewCLUE: A Chinese Few-Shot Learning Evaluation Benchmark[OL]. arXiv Preprint, arXiv:2107.07498.
[28]	Hu B T, Chen Q C, Zhu F Z. LCSTS: A Large Scale Chinese Short Text Summarization Dataset[C]// Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. 2015: 1967-1972.
[29]	Hua L F, Wan X J, Li L. Overview of the NLPCC 2017 Shared Task: Single Document Summarization[C]// Proceedings of National CCF Conference on Natural Language Processing and Chinese Computing. Springer International Publishing, 2018: 942-947.
[30]	Cui Y M, Liu T, Che W C, et al. A Span-Extraction Dataset for Chinese Machine Reading Comprehension[C]// Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019: 5883-5889.
[31]	Shao C C, Liu T, Lai Y, et al. DRCD: A Chinese Machine Reading Comprehension Dataset[OL]. arXiv Preprint, arXiv:1806.00920.
[32]	Sun K, Yu D, Yu D, et al. Investigating Prior Knowledge for Challenging Chinese Machine Reading Comprehension[J]. Transactions of the Association for Computational Linguistics, 2020, 8: 141-155. doi: 10.1162/tacl_a_00305
[33]	Li P, Li W, He Z Y, et al. Dataset and Neural Recurrent Sequence Labeling Model for Open-Domain Factoid Question Answering[OL]. arXiv Preprint, arXiv:1607.06275.
[34]	Lin C Y. ROUGE: A Package for Automatic Evaluation of Summaries[C]// Proceedings of the Workshop on Text Summarization Branches Out (WAS2004). 2004.

[1]	赵朝阳, 朱贵波, 王金桥. ChatGPT给语言大模型带来的启示和多模态大模型新的发展思路^*[J]. 数据分析与知识发现, 2023, 7(3): 26-35.
[2]	张智雄, 于改红, 刘熠, 林歆, 张梦婷, 钱力. ChatGPT对文献情报工作的影响^*[J]. 数据分析与知识发现, 2023, 7(3): 36-42.
[3]	钱力, 刘熠, 张智雄, 李雪思, 谢靖, 许钦亚, 黎洋, 管铮懿, 李西雨, 文森. ChatGPT的技术基础分析^*[J]. 数据分析与知识发现, 2023, 7(3): 6-15.
[4]	欧桂燕, 庞娜, 吴江. 专利审查周期影响因素研究——以中国人工智能领域为例*[J]. 数据分析与知识发现, 2022, 6(8): 20-30.
[5]	叶瀚,孙海春,李欣,焦凯楠. 融合注意力机制与句向量压缩的长文本分类模型[J]. 数据分析与知识发现, 2022, 6(6): 84-94.
[6]	景慎旗, 赵又霖. 基于医学领域知识和远程监督的医学实体关系抽取研究^*[J]. 数据分析与知识发现, 2022, 6(6): 105-114.
[7]	宋若璇,钱力,杜宇. 基于科技论文中未来工作句集的学术创新构想话题自动生成方法研究^*[J]. 数据分析与知识发现, 2021, 5(5): 10-20.
[8]	王义真,欧石燕,陈金菊. 民事裁判文书两阶段式自动摘要研究^*[J]. 数据分析与知识发现, 2021, 5(5): 104-114.
[9]	吕学强,罗艺雄,李家全,游新冬. 中文专利侵权检测研究综述^*[J]. 数据分析与知识发现, 2021, 5(3): 60-68.
[10]	沈卓,李艳. 基于PreLM-FT细粒度情感分析的餐饮业用户评论挖掘[J]. 数据分析与知识发现, 2020, 4(4): 63-71.
[11]	陆伟, 罗梦奇, 丁恒, 李信. 深度学习图像标注与用户标注比较研究^*[J]. 数据分析与知识发现, 2018, 2(5): 1-10.
[12]	巴志超,李纲,朱世伟. 基于语义网络的研究兴趣相似性度量方法^*[J]. 现代图书情报技术, 2016, 32(4): 81-90.
[13]	黄孝喜, 张华, 陆蓓, 王荣波, 吴铤. 一种基于词语抽象度的汉语隐喻识别方法[J]. 现代图书情报技术, 2015, 31(4): 34-40.
[14]	黄崑,符绍宏. 自动分词技术及其在信息检索中应用的研究[J]. 现代图书情报技术, 2001, 17(3): 26-29.
[15]	王咏,倪波,丁尉,承斌. 20世纪计算机软件技术的发展——IT技术世纪回眸之二[J]. 现代图书情报技术, 2000, 16(6): 6-9.

Viewed

Full text

Abstract

Cited

Shared

Discussed