Generating Chinese Abstracts with Content and Image Features
Quan Ankun1,2,Li Honglian1(),Zhang Le2,Lyu Xueqiang2
1School of Information & Communication Engineering, Beijing Information Science and Technology University, Beijing 100101, China 2Beijing Key Laboratory of Internet Culture and Digital Dissemination Research, Beijing Information Science and Technology University, Beijing 100101, China
[Objective] This paper proposes a new Chinese abstract generation method integrating content and image features. It aims to improve the performance of existing methods based on text features. [Methods] First, we used the BERT to extract text features and used ResNet to extract image features. Then, we utilized these features to complement and validate each other. Third, we fused the two modal features with the attention mechanism. Finally, we inputted the fused features into a pointer generation network to generate higher-quality Chinese abstracts. [Results] Compared to models solely relying on single text modality, the proposed method showed improvements of 1.9%, 1.3%, and 1.4% on ROUGE-1, ROUGE-2, and ROUGE-L metrics, respectively. [Limitations] The experimental data were primarily retrieved from the news domain, and the model’s effectiveness in other fields remains to be verified. [Conclusions] Incorporating image information allows the fused features to retain more important information. It helps the model identify the key content better and makes the generated abstracts more comprehensive and readable.
全安坤, 李红莲, 张乐, 吕学强. 融合内容和图片特征的中文摘要生成方法研究*[J]. 数据分析与知识发现, 2024, 8(3): 110-119.
Quan Ankun, Li Honglian, Zhang Le, Lyu Xueqiang. Generating Chinese Abstracts with Content and Image Features. Data Analysis and Knowledge Discovery, 2024, 8(3): 110-119.
(Ming Tuosiyu, Chen Hongchang. Research Progress and Trend of Text Summarization[J]. Chinese Journal of Network and Information Security, 2018, 4(6): 1-10.)
[2]
何丽. 基于多模态神经网络的图文摘要生成方法研究[D]. 北京: 北京邮电大学, 2021.
[2]
(He Li. Research on Method of Text-Image Summarization Based on Multimodal Neural Network[D]. Beijing: Beijing University of Posts and Telecommunications, 2021.)
[3]
See A, Liu P J, Manning C D. Get to the Point: Summarization with Pointer-Generator Networks[C]// Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics. 2017: 1073-1083.
(Tan Jinyuan, Diao Yufeng, Qi Ruihua, et al. Automatic Summary Generation of Chinese News Text Based on BERT-PGN Model[J]. Journal of Computer Applications, 2021, 41(1): 127-132.)
doi: 10.11772/j.issn.1001-9081.2020060920
(Li Jinpeng, Zhang Chuang, Chen Xiaojun, et al. Survey on Automatic Text Summarization[J]. Journal of Computer Research and Development, 2021, 58(1): 1-21.)
[6]
Mihalcea R, Tarau P. TextRank: Bringing Order into Text[C]// Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing. 2004: 404-411.
(Cheng Qikai, Wang Jiamin, Lu Wei. Discovering Domain Vocabularies Based on Citation Co-word Network[J]. Data Analysis and Knowledge Discovery, 2019, 3(6): 57-65.)
[8]
Sutskever I, Vinyals O, Le Q V. Sequence to Sequence Learning with Neural Networks[C]// Proceedings of the 27th International Conference on Neural Information Processing Systems. 2014: 3104-3112.
[9]
Shi T, Keneshloo Y, Ramakrishnan N, et al. Neural Abstractive Text Summarization with Sequence-to-Sequence Models[J]. ACM Transactions on Data Science, 2021, 2(1): 1-37.
[10]
Devlin J, Chang M W, Lee K, et al. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding[C]// Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics:Human Language Technologies. 2019: 4171-4186.
(Liu Zeyu, Ma Longlong, Wu Jian, et al. Chinese Image Captioning Method Based on Multimodal Neural Network[J]. Journal of Chinese Information Processing, 2017, 31(6): 162-171.)
[12]
陈祥. 基于多模态数据的文本摘要生成研究[D]. 成都: 电子科技大学, 2020.
[12]
(Chen Xiang. Research on Text Abstraction Generation Based on Multimodal Data[D]. Chengdu: University of Electronic Science and Technology of China, 2020.)
[13]
Li H R, Zhu J N, Ma C, et al. Multi-modal Summarization for Asynchronous Collection of Text, Image, Audio and Video[C]// Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. 2017: 1092-1102.
[14]
Li H R, Zhu J N, Liu T S, et al. Multi-modal Sentence Summarization with Modality Attention and Image Filtering[C]// Proceedings of the 27th International Joint Conference on Artificial Intelligence. 2018: 4152-4158.
[15]
Li M Z, Chen X Y, Gao S, et al. VMSMO: Learning to Generate Multimodal Summary for Video-Based News Articles[C]// Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing. 2020: 9360-9369.
[16]
Vaswani A, Shazeer N, Parmar N, et al. Attention is All You Need[C]// Proceedings of the 31st International Conference on Neural Information Processing Systems. 2017: 6000-6010.
(Liu Wenbin, He Yanqing, Wu Zhenfeng, et al. Sentence Alignment Method Based on BERT and Multi-similarity Fusion[J]. Data Analysis and Knowledge Discovery, 2021, 5(7): 48-58.)
[18]
Chen Y H. Convolutional Neural Network for Sentence Classification[D]. Waterloo: University of Waterloo, 2015.
[19]
Philipp G, Song D, Carbonell J G. The Exploding Gradient Problem Demystified-Definition, Prevalence, Impact, Origin, Tradeoffs, and Solutions[OL]. arXiv Preprint, arXiv: 1712.05577.
[20]
Bahdanau D, Cho K H, Bengio Y. Neural Machine Translation by Jointly Learning to Align and Translate[OL]. arXiv Preprint, arXiv: 1409.0473.
(Deng Zhenrong, Tang Yuanyu, Yang Rui, et al. Summarization Algorithm Based on Key Words and Pointer Generation Network[J]. Computer Systems and Applications, 2022, 31(11): 246-253.)
[22]
Lin C Y. ROUGE: A Package for Automatic Evaluation of Summaries[C]// Proceedings of the Workshop on Text Summarization Branches Out, Post-Conference Workshop of ACL 2004. 2004: 74-81.
