Please wait a minute...
Data Analysis and Knowledge Discovery  2023, Vol. 7 Issue (9): 100-113    DOI: 10.11925/infotech.2096-3467.2022.0854
Current Issue | Archive | Adv Search |
Studying Literature Reading Concentration Based on Multi-angle Facial Features
Liu Yang,Zhu Xuefang()
School of Information Management, Nanjing University, Nanjing 210023, China
Download: PDF (2837 KB)   HTML ( 11 ) 
Export: BibTeX | EndNote (RIS)      
Abstract  

[Objective] The concentration of literature reading is mainly evaluated by manual methods or eye-tracking techniques. This paper uses computer vision technology to automatically detect and receive real-time feedback from the concentration evaluation, which also improves the application of intelligent technology in smart knowledge service. [Methods] First, we detected the head postures of the readers by their vertical and horizontal rotation angles. Then, we scored their fatigue and emotion with the closing eyes or yawning status. Third, we decided the readers’ sentiment based on these expression recognition results. Fourth, we applied the fuzzy comprehensive evaluation algorithm to determine the weight of relevant factors. Finally, we integrated the scores to obtain the reader’s concentration status at different reading processes. [Results] We applied the new model to the actual reading scenes to evaluate the reading concentration of head tilt, fatigue, and negative emotion, and the results were 26.3%, 25.2%, and 6.8% lower than the normal state, respectively. [Limitations] When the literature reading video showed blurred facial features, the detection accuracy was unsatisfactory, which needs improvement. There are also some extreme reading instances to be optimized. [Conclusions] The proposed model can adjust reading strategies and help libraries optimize collection development strategies.
Key words： Literature Reading      Concentration Evaluation      Multi-angle Facial Features      Computer Vision      Fuzzy Comprehensive Evaluation     
Received: 15 August 2022      Published: 24 October 2023
ZTFLH:  G250  
  G350  
Fund:The National Social Science Fund of China(22BTQ017)
Corresponding Authors: Zhu Xuefang，ORCID： 0000-0002-8244-5999，E-mail： xfzhu@nju.edu.cn。   
Service
E-mail this article
Add to my bookshelf
Add to citation manager
E-mail Alert
RSS
Articles by authors
Yang Liu
Xuefang Zhu

Cite this article:

Liu Yang, Zhu Xuefang. Studying Literature Reading Concentration Based on Multi-angle Facial Features. Data Analysis and Knowledge Discovery, 2023, 7(9): 100-113.

URL:

https://manu44.magtech.com.cn/Jwk_infotech_wk3/EN/10.11925/infotech.2096-3467.2022.0854     OR     https://manu44.magtech.com.cn/Jwk_infotech_wk3/EN/Y2023/V7/I9/100

Roadmap for the Implementation of the Model
Structure of the YOLOv5 Model
The Effect of AP Iteration of Face Recognition Model
Schematic Diagram of Facial Posture
Schematic Diagram of Key Facial Feature Points
Schematic Diagram of the Key Feature Points of the Left Eye
Schematic Diagram of Monocular Correlation Distance
Accuracy at Different Eye Closure Thresholds
Diagram of the Key Points of the Mouth
Emotion Recognition Flowchart
VGGNet16 Structure Diagram
真实类 预测类
惊讶 开心 正常 伤心 厌恶 害怕 生气
惊讶 83 5 2 2 0 7 1
开心 2 88 6 2 0 1 1
正常 2 7 66 16 0 3 6
伤心 2 4 13 65 0 8 8
厌恶 2 0 0 11 61 4 22
害怕 7 3 9 20 0 53 8
生气 3 4 7 14 1 7 64
Expression Recognition Confusion Matrix Based on VGGNet16（%）
第一层因素 第二层因素
序号 指标名称 序号 指标名称
1 抬（低）头转动平均评分 1 头部姿态
2 面部左（右）转动平均评分
1 眼部相关疲劳度评分 2 疲劳度
2 嘴部相关疲劳度评分
1 惊讶表情频率 3 人脸表情
2 开心表情频率
3 正常表情频率
4 伤心表情频率
5 厌恶表情频率
6 害怕表情频率
7 生气表情频率
The Indicator System of the Concentration Evaluation Model
The Calculation Process of the Head Posture Composite Index
The Calculation Process of the Fatigue Composite Index
The Calculation Process of the Face Expression Composite Index
Hierarchy Based on the Model
n 1 2 3 4 5 6 7 8 9 10 11
R I 0 0 0.58 0.90 1.12 1.24 1.32 1.41 1.45 1.49 1.51
Stochastic Consistency Index R I
第一层因素 第二层因素
序号 权重 指标名称 序号 权重 指标名称
1 * 抬（低）头转动平均评分 1 0.637 头部姿态
2 面部左（右）转动平均评分
1 1.000 眼部相关疲劳度评分 2 0.258 疲劳度
2 1.000 嘴部相关疲劳度评分
1 0.374 惊讶表情频率 3 0.105 人脸表情
2 0.248 开心表情频率
3 0.171 正常表情频率
4 0.099 伤心表情频率
5 0.054 厌恶表情频率
6 0.027 害怕表情频率
7 0.027 生气表情频率
The Indicator System and Corresponding Weights of Literature Reading Concentration Model
The Mean Values of Literature Reading Concentration Scores in Different States Among 15 Testees
[1] 孙帅成, 刘瑞明. 专注度识别研究综述[J]. 科技资讯, 2021, 19(4):6-8, 39.
[1] (Sun Shuaicheng, Liu Ruiming. A Review of Research on Attentiveness Recognition[J]. Science & Technology Information, 2021, 19(4):6-8,39.)
[2] 石奕. 基于人脸识别技术的学生课堂专注度评价模型研究[D]. 武汉: 华中师范大学, 2020.
[2] (Shi Yi. Research on Evaluation Model of Classroom Attention of Students Based on Face Recognition Technology[D]. Wuhan: Central China Normal University, 2020.)
[3] 李文倩. 面向远程教育的学习专注度识别方法研究[D]. 杭州: 杭州电子科技大学, 2018.
[3] (Li Wenqian. Study on Learning Concentration Recognition Method for Distance Education[D]. Hangzhou: Hangzhou Dianzi University, 2018.)
[4] 熊碧辉. 在线学习中融合视线分析的注意力检测方法的研究和应用[D]. 杭州: 杭州电子科技大学, 2018.
[4] (Xiong Bihui. Research and Application of Attention Detection Method with Eye Gaze Analysis in Online Learning[D]. Hangzhou: Hangzhou Dianzi University, 2018.)
[5] Chen X, Xie L, He S, et al. Research on a Driver Fatigue State Detection System[C]// Proceedings of the 13th International Conference on Anti-counterfeiting, Security, and Identification (ASID). IEEE, 2019:253-257.
[6] 卫旭骏, 梁承姬, 潘洋, 等. 基于眼动分析的不同光环境下岸桥司机的专注度[J]. 大连海事大学学报, 2015, 41(3):93-99.
[6] (Wei Xujun, Liang Chengji, Pan Yang, et al. Quayside Crane Driver’s Concentration Research under Different Luminance Environment Based on Eye Movement Analysis[J]. Journal of Dalian Maritime University, 2015, 41(3):93-99.)
[7] 王淑珍, 于园园, 帅静. 目标监控岗位员工选拔中持续性注意力的测评[J]. 浙江大学学报(理学版), 2015, 42(4):489-493.
[7] Wang Shuzhen, Yu Yuanyuan, Shuai Jing. Measurements on the Sustained Attention During the Employee Selection for the Supervisory Control[J]. Journal of Zhejiang University(Science Edition), 2015, 42(4):489-493.)
[8] Zhao L, Zhang Y, Zhang C. Does Attention Mechanism Possess the Feature of Human Reading? A Perspective of Sentiment Classification Task[J]. Aslib Journal of Information Management, 2023, 75(1):20-43.
doi: 10.1108/AJIM-12-2021-0385
[9] Zhang Y, Zhang C. Enhancing Keyphrase Extraction from Microblogs Using Human Reading Time[J]. Journal of the Association for Information Science and Technology, 2021, 72(5):611-626.
doi: 10.1002/asi.v72.5
[10] 楊雅婷, 陳奕樺. 認知策略,心情沉靜狀態,課室專注力與學業成就之關聯:以台灣一所高級職業學校為例[J]. 教育學報, 2014, 42(2):23-50.
[10] (Yang Yating, Chan Ihua. Examining the Relationship Between Cognitive Strategies, Calmness,Classroom Concentration, and Academic Achievement:Taking a Vocational School in Taiwan as An Example[J]. Education Journal, 2014, 42(2):23-50.)
[11] 钟马驰, 张俊朗, 蓝扬波, 等. 基于人脸检测和模糊综合评判的在线教育专注度研究[J]. 计算机科学, 2020, 47(suppl.2):196-203.
[11] (Zhong Machi, Zhang Junlang, Lan Yangbo, et al. Study on Online Education Focus Degree Based on Face Detection and Fuzzy Comprehensive Evaluation[J]. Computer Science, 2020, 47(suppl.2):196-203.)
[12] 李月龙, 靳彦, 汪剑鸣, 等. 人脸特征点提取方法综述[J]. 计算机学报, 2016, 39(7):1356-1374.
[12] (Li Yuelong, Jin Yan, Wang Jianming, et al. Face Feature Points Extraction:A Review[J]. Chinese Journal of Computers, 2016, 39(7):1356-1374.)
[13] 许德刚, 王露, 李凡. 深度学习的典型目标检测算法研究综述[J]. 计算机工程与应用, 2021, 57(8):10-25.
doi: 10.3778/j.issn.1002-8331.2012-0449
[13] (Xu Degang, Wang Lu, Li Fan. Review of Typical Object Detection Algorithms for Deep Learning[J]. Computer Engineering and Applications, 2021, 57(8):10-25.)
doi: 10.3778/j.issn.1002-8331.2012-0449
[14] 王鹏, 王玉林, 焦博文, 等. 基于YOLOv5的道路目标检测算法研究[J]. 计算机工程与应用, 2023, 59(1):117-125.
doi: 10.3778/j.issn.1002-8331.2206-0316
[14] (Wang Peng, Wang Yulin, Jiao Bowen, et al. Research on Road Target Detection Algorithm Based on YOLOv5[J]. Computer Engineering and Applications, 2023, 59(1):117-125.)
doi: 10.3778/j.issn.1002-8331.2206-0316
[15] Yang S, Luo P, Loy C C, et al. Wider Face: A Face Detection Benchmark[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2016: 5525-5533.
[16] 刘兆丰. Dlib在人脸识别技术中的运用[J]. 电子制作, 2020(21):39-41,7.
[16] (Liu Zhaofeng. Application of Dlib in Face Recognition Technology[J]. Practical Electronics, 2020(21):39-41,7.)
[17] Lepetit V, Moreno-Noguer F, Fua P. EPnP: An Accurate O(n) Solution to the PnP Problem[J]. International Journal of Computer Vision, 2009, 81(2):155-166.
doi: 10.1007/s11263-008-0152-6
[18] Gourier N, Hall D, Crowley J L. Estimating Face Orientation from Robust Detection of Salient Facial Structures[C]// FG Net Workshop on Visual Observation of Deictic Gestures, Cambridge, UK. 2004.
[19] 李永杰, 周桂红, 刘博. 基于YOLOv3模型的人脸检测与头部姿态估计融合算法[J]. 广西师范大学学报(自然科学版), 2022, 40(3):95-103.
[19] Li Yongjie, Zhou Guihong, Liu Bo. Fusion Algorithm of Face Detection and Head Pose Estimation Based on YOLOv3 Model[J]. Journal of Guangxi Normal University(Natural Science Edition), 2022, 40(3):95-103.)
[20] Song F, Tan X, Chen S, et al. A Literature Survey on Robust and Efficient Eye Localization in Real-life Scenarios[J]. Pattern Recognition, 2013, 46(12):3157-3173.
doi: 10.1016/j.patcog.2013.05.009
[21] Huang G B, Mattar M, Berg T, et al. Labeled Faces in the Wild: A Database for Studying Face Recognition in Unconstrained Environments[C]// Workshop on Faces in‘Real-Life’Images: Detection, Alignment, and Recognition, Marseille, France. 2008.
[22] Abtahi S, Omidyeganeh M, Shirmohammadi S, et al. YawDD: A Yawning Detection Dataset[C]// Proceedings of the 5th ACM Multimedia Systems Conference. 2014: 24-28.
[23] 李萌萌, 马力, 贾宇峰. 模式搜索星型生成对抗网络下的面部表情生成[J]. 传感器与微系统, 2022, 41(7):147-151.
[23] (Li Mengmeng, Ma Li, Jia Yufeng. Facial Expression Generation Under Mode Seeking StarGAN[J]. Transducer and Microsystem Technologies, 2022, 41(7):147-151 )
[24] 冯墨女, 刘晓明. 大学生认知抑制与情绪和行动控制风格的关系[J]. 中国心理卫生杂志, 2019, 33(11):862-867.
[24] (Feng Monü, Liu Xiaoming. Relation of Cognitive Inhibition to Emotion and Action Control Style in College Students[J]. Chinese Mental Health Journal, 2019, 33(11):862-867.)
[25] Simonyan K, Zisserman A.Very Deep Convolutional Networks for Large-scale Image Recognition[OL].arXiv Preprint,arXiv:1409.1556.
[26] Li S, Deng W. Deep Facial Expression Recognition: A Survey[J]. IEEE Transactions on Affective Computing, 2022, 13(3):1195-1215.
doi: 10.1109/TAFFC.2020.2981446
[27] Goodfellow I J, Erhan D, Carrier P L, et al. Challenges in Representation Learning: A Report on Three Machine Learning Contests[C]// Proceedings of International Conference on Neural Information Processing. Springer, 2013: 117-124.
[28] 金菊良, 魏一鸣, 丁晶. 基于改进层次分析法的模糊综合评价模型[J]. 水利学报, 2004(3):65-70.
[28] (Jin Juliang, Wei Yiming, Ding Jing. Fuzzy Comprehensive Evaluation Model Based on Improved Analytic Hierarchy Process[J]. Journal of Hydraulic Engineering, 2004(3):65-70.)
[29] 王俊岭, 熊玉华, 张现国, 等. 基于AHP-模糊综合评价法的城市排水管网状态和运行效能评价——以淮安市淮安区为例[J]. 环境工程技术学报, 2022, 12(4):1162-1169.
[29] (Wang Junling, Xiong Yuhua, Zhang Xianguo, et al. Evaluation of the State and Operational Effectiveness of Urban Drainage Pipe Network Based AHP-Fuzzy Comprehensive Evaluation Method:Taking Huai’An District of Huai’An City as An Example[J]. Journal of Environmental Engineering Technology, 2022, 12(4):1162-1169.)
[30] Wierwille W W, Wreggit S S, Kirn C L, et al. Research on Vehicle-based Driver Status/Performance Monitoring; Development, Validation, and Refinement of Algorithms for Detection of Driver Drowsiness[R]. National Highway Traffic Safety Administratlon, 1994.
[31] 郭金玉, 张忠彬, 孙庆云. 层次分析法的研究与应用[J]. 中国安全科学学报, 2008, 18(5):148-153.
[31] (Guo Jinyu, Zhang Zhongbin, Sun Qingyun. Study and Applications of Analytic Hierarchy Process[J]. China Safety Science Journal, 2008, 18(5):148-153.)
[32] 陈道想, 林鹏, 丁鹏, 等. 基于群层次分析法的振冲碎石桩填料方法比选[J]. 清华大学学报(自然科学版), 2022, 62(12):1915-1921.
[32] Chen Daoxiang, Lin Peng, Ding Peng, et al. Vibro-stone Column Filling Schemes Based on Group AHP[J]. Journal of Tsinghua University(Science and Technology), 2022, 62(12):1915-1921.)
[1] Zhou Zeyu, Wang Hao, Zhang Xiaoqin, Tao Fao, Ren Qiutong. Classification Model for Chinese Traditional Embroidery Based on Xception-TD[J]. 数据分析与知识发现, 2022, 6(2/3): 338-347.
[2] Wu Dan,Lu Liuxing. Impacts of Mobile Tools on Students’ Academic Reading Efficiency[J]. 数据分析与知识发现, 2017, 1(1): 64-72.
[3] Yin Xihong, Qiao Xiaodong, Zhang Yunliang, Li Guoshuang. Fuzzy Classification Method Based on Particle Swarm Optimization and Fuzzy Comprehensive Evaluation[J]. 现代图书情报技术, 2015, 31(9): 46-51.
[4] Ma Xiaoying,Han Shufen,Sun Duoqing . The Quasi-entropy Weight Fuzzy Comprehensive Evaluation of Library Readers’ Satisfaction[J]. 现代图书情报技术, 2008, 24(6): 95-99.
[5] Pang Qinghua . A Study on the Method of Comprehensive Evaluation for Library Website[J]. 现代图书情报技术, 2006, 1(6): 52-54.
  Copyright © 2016 Data Analysis and Knowledge Discovery   Tel/Fax:(010)82626611-6626，82624938   E-mail:jishu@mail.las.ac.cn