1National Science Library, Chinese Academy of Sciences, Beijing 100190, China 2Department of Information Resources Management, School of Economics and Management, University of Chinese Academy of Sciences, Beijing 100190, China 3Key Laboratory of New Publishing and Knowledge Services for Scholarly Journals, Beijing 100190, China
[Objective] Review and analyze the corpus, algorithms and models related to ChatGPT, and provide a systematic reference for peer research. [Methods] This paper systematically reviewed the relevant literature and materials since the release of GPT-3. We depict the overall architecture of ChatGPT technology, and explain and analyze the models, algorithms, and principles behind it. [Results] This paper restores the technical details that support ChatGPT functionality based on limited information through literature research. Rationalizing the overall technical architecture diagram of ChatGPT and explaining each technical component of it. The algorithmic principles and model composition of each technical component of ChatGPT is analyzed at three levels: the corpus system, the pre-training algorithm and model, and the fine-tuning algorithm and model. [Limitations] The investigation of the literature related to ChatGPT inevitably has omissions, and the interpretation of some technical contents is not deep enough. Some contents inferred by the authors may be incorrect. [Conclusions] The breakthrough in the application of ChatGPT technology is the result of continuous accumulation through iterative training of corpora, models and algorithms, as well as the effective combination and integration of various algorithmic models.
钱力, 刘熠, 张智雄, 李雪思, 谢靖, 许钦亚, 黎洋, 管铮懿, 李西雨, 文森. ChatGPT的技术基础分析*[J]. 数据分析与知识发现, 2023, 7(3): 6-15.
Qian Li, Liu Yi, Zhang Zhixiong, Li Xuesi, Xie Jing, Xu Qinya, Li Yang, Guan Zhengyi, Li Xiyu, Wen Sen. An Analysis on the Basic Technologies of ChatGPT. Data Analysis and Knowledge Discovery, 2023, 7(3): 6-15.
Zhang Zhixiong, Qian Li, Xie Jing, et al. The Impact of ChatGPT on Scientific Research and Library & Information Service[R]. Beijing: National Science Library, Chinese Academy of Sciences, National Science and Technology Digital Library, 2023.)
[3]
Zhou C, Li Q, Li C, et al. A Comprehensive Survey on Pretrained Foundation Models: A History from BERT to ChatGPT[OL]. arXiv Preprint, arXiv:2302.09419.
[4]
Cao Y H, Li S Y, Liu Y X, et al. A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPT[OL]. arXiv Preprint, arXiv:2303.04226.
[5]
Brown T B, Mann B, Ryder N, et al. Language Models are Few-Shot Learners[OL]. arXiv Preprint, arXiv:2005.14165.
[6]
Neelakantan A, Xu T, Puri R, et al. Text and Code Embeddings by Contrastive Pre-Training[OL]. arXiv Preprint, arXiv: 2201.10005.
[7]
Ouyang L, Wu J, Jiang X, et al. Training Language Models to Follow Instructions with Human Feedback[OL]. arXiv Preprint, arXiv:2203.02155.
[8]
Wang F Y, Miao Q, Li X, et al. What does ChatGPT Say: The DAO from Algorithmic Intelligence to Linguistic Intelligence[J]. IEEE/CAA Journal of Automatica Sinica, 2023, 10(3): 575-579.
doi: 10.1109/JAS.2023.123486
[9]
Chen M, Tworek J, Jun H, et al. Evaluating Large Language Models Trained on Code[OL]. arXiv Preprint, arXiv:2107.03374.
[10]
Thompson A D. What’s in My AI?[EB/OL]. [2023-03-12]. https://lifearchitect.ai/whats-in-my-ai/.
[11]
OpenAI, Aligning Language Models to Follow Instructions[EB/OL]. [2023-03-12]. https://openai.com/blog/instruction-following/.
[12]
Vaswani A, Shazeer N, Parmar N, et al. Attention is All You Need[OL]. arXiv Preprint, arXiv:1706.03762v2.
Neelakantan A, Weng L L, Power B, et al. Introducing Text and Code Embeddings[EB/OL]. [2023-03-12]. https://openai.com/blog/introducing-text-and-code-embeddings.
[15]
van den Oord A, Li Y Z, Vinyals O. Representation Learning with Contrastive Predictive Coding[OL]. arXiv Preprint, arXiv:1807.03748.
[16]
OpenAI. What are Embeddings?[EB/OL]. [2023-03-12]. https://platform.openai.com/docs/guides/embeddings/what-are-embeddings.
Lachaux M A, Roziere B, Chanussot L, et al. Unsupervised Translation of Programming Languages[OL]. arXiv Preprint, arXiv:2006.03511.
[19]
Kulal S, Pasupat P, Chandra K, et al. SPoC: Search-based Pseudocode to Code[C]// Proceedings of the 33rd Conference on Neural Information Processing Systems (NeurIPS 2019). 2019.
OpenAI. API Guide Text Completion Inserting Text[EB/OL]. [2023-03-13]. https://platform.openai.com/docs/guides/completion/inserting-text.
[22]
OpenAI. API Guide Text Completion Editing Text[EB/OL]. [2023-03-13]. .
[23]
Thompson A D. GPT-3.5 + ChatGPT: An Illustrated Overview[EB/OL]. [2023-03-12]. https://lifearchitect.ai/chatgpt/.
[24]
Fu Y. How does GPT Obtain Its Ability? Tracing Emergent Abilities of Language Models to Their Sources[EB/OL]. [2023-03-12]. https://yaofu.notion.site/How-does-GPT-Obtain-its-Ability-Tracing-Emergent-Abilities-of-Language-Models-to-their- Sources-b9a57ac0fcf74f30a1ab9e3e36fa1dc1.
[25]
Wei J, Bosma M, Zhao V Y, et al. Finetuned Language Models are Zero-Shot Learners [OL]. arXiv Preprint, arXiv:2109.01652.
[26]
Schulman J, Klimov O, Wolski F, et al. Proximal Policy Optimization[EB/OL]. [2023-03-12]. https://openai.com/research/openai-baselines-ppo.
[27]
Joyce J M. Kullback-Leibler Divergence[A]// International Encyclopedia of Statistical Science[M]. Springer, 2011: 720-722.
[28]
Gao L, Schulman J, Hilton J. Scaling Laws for Reward Model Overoptimization[EB/OL]. [2023-03-12]. https://openai.com/research/scaling-laws-for-reward-model-overoptimization.
[29]
OpenAI. Model Index for Researchers[EB/OL]. [2023-03-12]. https://platform.openai.com/docs/model-index-for-researchers.
