|
|
|
| Generating Chinese Abstracts with Content and Image Features |
Quan Ankun1,2,Li Honglian1( ),Zhang Le2,Lyu Xueqiang2 |
1School of Information & Communication Engineering, Beijing Information Science and Technology University, Beijing 100101, China
2Beijing Key Laboratory of Internet Culture and Digital Dissemination Research, Beijing Information Science and Technology University, Beijing 100101, China |
|
|
|
|
Abstract [Objective] This paper proposes a new Chinese abstract generation method integrating content and image features. It aims to improve the performance of existing methods based on text features. [Methods] First, we used the BERT to extract text features and used ResNet to extract image features. Then, we utilized these features to complement and validate each other. Third, we fused the two modal features with the attention mechanism. Finally, we inputted the fused features into a pointer generation network to generate higher-quality Chinese abstracts. [Results] Compared to models solely relying on single text modality, the proposed method showed improvements of 1.9%, 1.3%, and 1.4% on ROUGE-1, ROUGE-2, and ROUGE-L metrics, respectively. [Limitations] The experimental data were primarily retrieved from the news domain, and the model’s effectiveness in other fields remains to be verified. [Conclusions] Incorporating image information allows the fused features to retain more important information. It helps the model identify the key content better and makes the generated abstracts more comprehensive and readable.
|
|
Received: 07 December 2022
Published: 16 May 2023
|
|
|
| Fund:National Natural Science Foundation of China(62171043);“Diligent Talents” Training Scheme Foundation of Beijing Information Science and Technology University(QXTCP B201908) |
|
Corresponding Authors:
Li Honglian，ORCID：0000-0002-0531-3650，E-mail：lihonglian@bistu.edu.cn。
|
| [1] |
明拓思宇, 陈鸿昶. 文本摘要研究进展与趋势[J]. 网络与信息安全学报, 2018, 4(6): 1-10.
|
| [1] |
(Ming Tuosiyu, Chen Hongchang. Research Progress and Trend of Text Summarization[J]. Chinese Journal of Network and Information Security, 2018, 4(6): 1-10.)
|
| [2] |
何丽. 基于多模态神经网络的图文摘要生成方法研究[D]. 北京: 北京邮电大学, 2021.
|
| [2] |
(He Li. Research on Method of Text-Image Summarization Based on Multimodal Neural Network[D]. Beijing: Beijing University of Posts and Telecommunications, 2021.)
|
| [3] |
See A, Liu P J, Manning C D. Get to the Point: Summarization with Pointer-Generator Networks[C]// Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics. 2017: 1073-1083.
|
| [4] |
谭金源, 刁宇峰, 祁瑞华, 等. 基于BERT-PGN模型的中文新闻文本自动摘要生成[J]. 计算机应用, 2021, 41(1): 127-132.
doi: 10.11772/j.issn.1001-9081.2020060920
|
| [4] |
(Tan Jinyuan, Diao Yufeng, Qi Ruihua, et al. Automatic Summary Generation of Chinese News Text Based on BERT-PGN Model[J]. Journal of Computer Applications, 2021, 41(1): 127-132.)
doi: 10.11772/j.issn.1001-9081.2020060920
|
| [5] |
李金鹏, 张闯, 陈小军, 等. 自动文本摘要研究综述[J]. 计算机研究与发展, 2021, 58(1): 1-21.
|
| [5] |
(Li Jinpeng, Zhang Chuang, Chen Xiaojun, et al. Survey on Automatic Text Summarization[J]. Journal of Computer Research and Development, 2021, 58(1): 1-21.)
|
| [6] |
Mihalcea R, Tarau P. TextRank: Bringing Order into Text[C]// Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing. 2004: 404-411.
|
| [7] |
程齐凯, 王佳敏, 陆伟. 基于引用共词网络的领域基础词汇发现研究[J]. 数据分析与知识发现, 2019, 3(6): 57-65.
|
| [7] |
(Cheng Qikai, Wang Jiamin, Lu Wei. Discovering Domain Vocabularies Based on Citation Co-word Network[J]. Data Analysis and Knowledge Discovery, 2019, 3(6): 57-65.)
|
| [8] |
Sutskever I, Vinyals O, Le Q V. Sequence to Sequence Learning with Neural Networks[C]// Proceedings of the 27th International Conference on Neural Information Processing Systems. 2014: 3104-3112.
|
| [9] |
Shi T, Keneshloo Y, Ramakrishnan N, et al. Neural Abstractive Text Summarization with Sequence-to-Sequence Models[J]. ACM Transactions on Data Science, 2021, 2(1): 1-37.
|
| [10] |
Devlin J, Chang M W, Lee K, et al. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding[C]// Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics:Human Language Technologies. 2019: 4171-4186.
|
| [11] |
刘泽宇, 马龙龙, 吴健, 等. 基于多模态神经网络的图像中文摘要生成方法[J]. 中文信息学报, 2017, 31(6): 162-171.
|
| [11] |
(Liu Zeyu, Ma Longlong, Wu Jian, et al. Chinese Image Captioning Method Based on Multimodal Neural Network[J]. Journal of Chinese Information Processing, 2017, 31(6): 162-171.)
|
| [12] |
陈祥. 基于多模态数据的文本摘要生成研究[D]. 成都: 电子科技大学, 2020.
|
| [12] |
(Chen Xiang. Research on Text Abstraction Generation Based on Multimodal Data[D]. Chengdu: University of Electronic Science and Technology of China, 2020.)
|
| [13] |
Li H R, Zhu J N, Ma C, et al. Multi-modal Summarization for Asynchronous Collection of Text, Image, Audio and Video[C]// Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. 2017: 1092-1102.
|
| [14] |
Li H R, Zhu J N, Liu T S, et al. Multi-modal Sentence Summarization with Modality Attention and Image Filtering[C]// Proceedings of the 27th International Joint Conference on Artificial Intelligence. 2018: 4152-4158.
|
| [15] |
Li M Z, Chen X Y, Gao S, et al. VMSMO: Learning to Generate Multimodal Summary for Video-Based News Articles[C]// Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing. 2020: 9360-9369.
|
| [16] |
Vaswani A, Shazeer N, Parmar N, et al. Attention is All You Need[C]// Proceedings of the 31st International Conference on Neural Information Processing Systems. 2017: 6000-6010.
|
| [17] |
刘文斌, 何彦青, 吴振峰, 等. 基于BERT和多相似度融合的句子对齐方法研究[J]. 数据分析与知识发现, 2021, 5(7): 48-58.
|
| [17] |
(Liu Wenbin, He Yanqing, Wu Zhenfeng, et al. Sentence Alignment Method Based on BERT and Multi-similarity Fusion[J]. Data Analysis and Knowledge Discovery, 2021, 5(7): 48-58.)
|
| [18] |
Chen Y H. Convolutional Neural Network for Sentence Classification[D]. Waterloo: University of Waterloo, 2015.
|
| [19] |
Philipp G, Song D, Carbonell J G. The Exploding Gradient Problem Demystified-Definition, Prevalence, Impact, Origin, Tradeoffs, and Solutions[OL]. arXiv Preprint, arXiv: 1712.05577.
|
| [20] |
Bahdanau D, Cho K H, Bengio Y. Neural Machine Translation by Jointly Learning to Align and Translate[OL]. arXiv Preprint, arXiv: 1409.0473.
|
| [21] |
邓珍荣, 汤园钰, 杨睿, 等. 基于关键词与指针生成网络的摘要生成算法[J]. 计算机系统应用, 2022, 31(11): 246-253.
|
| [21] |
(Deng Zhenrong, Tang Yuanyu, Yang Rui, et al. Summarization Algorithm Based on Key Words and Pointer Generation Network[J]. Computer Systems and Applications, 2022, 31(11): 246-253.)
|
| [22] |
Lin C Y. ROUGE: A Package for Automatic Evaluation of Summaries[C]// Proceedings of the Workshop on Text Summarization Branches Out, Post-Conference Workshop of ACL 2004. 2004: 74-81.
|
|
Viewed |
|
|
|
Full text
|
|
|
|
|
Abstract
|
|
|
|
|
Cited |
|
|
|
|
| |
Shared |
|
|
|
|
| |
Discussed |
|
|
|
|
