Recognition of Emotions and Analysis of Emotional Changes in Chinese Folk Songs
Zhao Meng1,2,Wang Hao1,2(),Li Xiaomin1,2
1School of Information Management, Nanjing University, Nanjing 210023,China 2Jiangsu Key Laboratory of Data Engineering and Knowledge Service, Nanjing 210023,China
[Objective] This paper aims to achieve the automatic recognition of rich emotions in Chinese folk songs and to explore their emotional context and fluctuation patterns digitally. [Methods] We adopted Hevner's emotion model in the field of music and introduced external Chinese knowledge for the semantic enhancement of emotion words. The automatic mapping of artificially labelled tags is then realized by semantic distance calculation. We constructed a multimodal multitag emotion recognition model (MMERM) that fuses features of lyrics and audios for automatic emotion recognition. The model is also transferred to recognize changes of emotions in songs, based on which statistical analysis and visualization of emotional context and fluctuation patterns can be conducted. [Results] The semantic enhancement and mapping effectively improve the concentration and differentiation of tags in emotion recognition. MMERM performs well on both complete songs and fragments, with a precision of 82.29%. Regularity analysis indicates a changing trend of lightness to sadness and sacredness from the beginning to the end of the songs. Furthermore, the fluctuation pattern of Chinese folk songs is found to differ remarkably from that of Western music. [Limitations] The information of folk songs is insufficient, and emotional characteristics under different temporal and spatial conditions are not analyzed. [Conclusions] This paper provides a new paradigm for the research of traditional music from the perspective of digital humanities.
赵萌, 王昊, 李晓敏. 中国民歌多情感识别及情感变化规律分析研究*[J]. 数据分析与知识发现, 2023, 7(7): 111-124.
Zhao Meng, Wang Hao, Li Xiaomin. Recognition of Emotions and Analysis of Emotional Changes in Chinese Folk Songs. Data Analysis and Knowledge Discovery, 2023, 7(7): 111-124.
(Zhang Tongtong. Exploring the Development Path of Excellent Chinese Traditional Folk Songs in the 14th Five-Year Plan Period[J]. Song of the Yellow River, 2022(2): 27-29.)
(Zhang Xi. Sacred and Secular: The Transmutation of the Lyrical Space of Zhuang Folk Songs from the Perspective of Emotional Expression[J]. Songs Bimonthly, 2020(6):76-80.)
[3]
郝婷婷. 陕北民歌《兰花花》的风格探析[D]. 西安: 西安音乐学院, 2018.
[3]
(Hao Tingting. Investigation and Analysis on the Musical Style of the Classical Folk Song: The Orchid Flower[D]. Xi'an: Xi'an Conservatory of Music, 2018.)
(Wang Hailong, Liu Lin, Ji Wenlu, et al. The Construction of Mongolian Music Resource Database[J]. Journal of Shaanxi Normal University (Natural Science Edition), 2021, 49(5): 109-116.)
(Kang Jian, Wang Hailong, Su Guibin, et al. Survey of Music Emotion Recognition[J]. Computer Engineering and Applications, 2022, 58(4): 64-72.)
doi: 10.3778/j.issn.1002-8331.2108-0538
[6]
Han D, Kong Y, Han J, et al. A Survey of Music Emotion Recognition[J]. Frontiers of Computer Science, 2022, 16(6): Article No. 166335.
(Yan Chengyi. Research on Multi-Label Music Emotion Based on Evidence Theory and Information Cell Model[D]. Hangzhou: Zhejiang University, 2012.)
[8]
Hevner K. Experimental Studies of the Elements of Expression in Music[J]. The American Journal of Psychology, 1936, 48(2): 246-268.
doi: 10.2307/1415746
[9]
刘涛. 音乐情感认知模型与交互技术研究[D]. 杭州: 浙江大学, 2006.
[9]
(Liu Tao. Research on Music Emotion Recognition Model and Interactive Technology[D]. Hangzhou: Zhejiang University, 2006.)
(Jiang Shengyi, Yang Yao, Liao Jingxin. Research of Building Chinese Musical Emotional Lexicon and Emotional Classification[J]. Computer Engineering and Applications, 2014, 50(24): 118-121.)
(Xing Baixi, Xie Linhai, Zhu Bohan, et al. Research on the Cross-Media Synesthesia Matching of Chinese Poetry and Folk Music Based on Emotional Characteristics[J]. Journal of Fudan University (Natural Science), 2020, 59(5): 558-564.)
(Guo Jiying, Liu Jingyu, Jiang Yujian. A Study on the Relationship Between Timbre Perception Features and Emotions in Musical Sounds[J]. Journal of Fudan University (Natural Science), 2021, 60(3): 347-352.)
[13]
Han E, Cha H. Audio Feature Extraction for Effective Emotion Classification[J]. IEIE Transactions on Smart Processing & Computing, 2019, 8(2): 100-107.
[14]
Li J, Han L, Li X, et al. An Evaluation of Deep Neural Network Models for Music Classification Using Spectrograms[J]. Multimedia Tools and Applications, 2022, 81(4): 4621-4647.
doi: 10.1007/s11042-020-10465-9
[15]
Gordon R L, Schön D, Magne C, et al. Words and Melody are Intertwined in Perception of Sung Words: EEG and Behavioral Evidence[J]. PLoS ONE, 2010, 5(3): e9889.
doi: 10.1371/journal.pone.0009889
[16]
Xu L, Sun Z, Wen X, et al. Using Machine Learning Analysis to Interpret the Relationship Between Music Emotion and Lyric Features[J]. PeerJ Computer Science, 2021, 7: e785.
doi: 10.7717/peerj-cs.785
pmid: 34901433
(Zhao Jian, Liu Huaping, Liang Xiaojing, et al. Multimodal Music Emotion Recognition Based on Knowledge Distillation and Transfer Learning[J]. Journal of Fudan University (Natural Science), 2021, 60(3): 309-314.)
[18]
Deng J J, Leung C H C. Dynamic Time Warping for Music Retrieval Using Time Series Modeling of Musical Emotions[J]. IEEE Transactions on Affective Computing, 2015, 6(2): 137-151.
doi: 10.1109/TAFFC.2015.2404352
[19]
Zhang Z, Xie L, Zhao Z. Temporal Integration of Emotion Perception for Cross-Cultural and Multi-Emotion Music[C]// Proceedings of 2021 International Conference on Culture-oriented Science & Technology (ICCST). IEEE, 2021: 198-203.
[20]
Hasanzadeh F, Annabestani M, Moghimi S. Continuous Emotion Recognition During Music Listening Using EEG Signals: A Fuzzy Parallel Cascades Model[J]. Applied Soft Computing, 2021, 101: 107028.
doi: 10.1016/j.asoc.2020.107028
[21]
杨媛. 数字音乐学研究综述[J]. 北方音乐, 2020(2): 4-5, 14.
[21]
(Yang Yuan. A Review of Digital Musicology Research[J]. Journal of the Northern Music, 2020(2): 4-5, 14.)
[22]
Yang Y H, Chen H H. Machine Recognition of Music Emotion: A Review[J]. ACM Transactions on Intelligent Systems and Technology (TIST), 2012, 3(3): Article No.40.
[23]
Devlin J, Chang M W, Lee K, et al. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding[OL]. arXiv Preprint, arXiv:1810.04805.
[24]
Liu G, Guo J. Bidirectional LSTM with Attention Mechanism and Convolutional Layer for Text Classification[J]. Neurocomputing, 2019, 337: 325-338.
doi: 10.1016/j.neucom.2019.01.078
[25]
Logan B. Mel Frequency Cepstral Coefficients for Music Modeling[C]// Proceedings of the International Symposium on Music Information Retrieval. 2000.
[26]
Baziotis C, Pelekis N, Doulkeridis C. DataStories at SemEval-2017 Task 4: Deep LSTM with Attention for Message-level and Topic-based Sentiment Analysis[C]// Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017). ACM Press, 2017: 747-754.
[27]
Zia T, Zahid U. Long Short-Term Memory Recurrent Neural Network Architectures for Urdu Acoustic Modeling[J]. International Journal of Speech Technology, 2019, 22(1): 21-30.
doi: 10.1007/s10772-018-09573-7
[28]
Pearl R, Reed L J. On the Rate of Growth of the Population of the United States Since 1790 and Its Mathematical Representation[J]. PNAS, 1920, 6(6): 275-288.
doi: 10.1073/pnas.6.6.275
pmid: 16576496
(Zhang Chiming, Wang Qingfeng, Liu Zhiqin, et al. Pulmonary Nodule Auxiliary Diagnosis Method Based on Deep Transfer Learning[J]. Computer Engineering, 2020, 46(1): 271-278.)
doi: 10.19678/j.issn.1000-3428.0053340
(Zhang Yi. A Probe into Similarities and Dissimilarities Between Chinese Music and Western Music[J]. Journal of Hainan Normal University (Social Sciences), 2012, 25(5): 142-144.)
