Please wait a minute...
Data Analysis and Knowledge Discovery  2023, Vol. 7 Issue (6): 123-133    DOI: 10.11925/infotech.2096-3467.2022.0485
Current Issue | Archive | Adv Search |
Evaluating Student Engagement with Deep Learning
Wang Nan1,2,Wang Qi1()
1School of Management Science and Information Engineering, Jilin University of Finance and Economics, Changchun 130117, China
2Business Big Data Research Center of Jilin Province, Changchun 130117, China
Download: PDF (2928 KB)   HTML ( 18 ) 
Export: BibTeX | EndNote (RIS)      
Abstract  

[Objective] This paper constructs an expression data set of engagement degrees and designs a joint evaluation model for students’ class engagement. It addresses the issues of lacking relevant expression data sets and the low accuracy of the existing models. [Methods] We collected data based on actual online classes and constructed an expression dataset suitable for engagement recognition. Then, we designed an improved VGG model to evaluate the dataset and recognize student engagement. Third, we combined the expression and face scores to establish a joint evaluation model for students’ engagement and calculated the tested students’ actual class engagement scores. [Results] We adjusted and verified the network structure through parameter tuning optimization for engagement expression recognition. The improved model VGG16+Dense+Dropout(lr=1e-5) had the highest accuracy among the four compared model architectures, reaching over 92%. The joint engagement score is more accurate for engagement evaluation than the single expression engagement score. [Limitations] We did not include more ablation studies in training the model; more research is needed to explore the deeper neural networks. [Conclusions] The dataset of W-AttLe is suitable for evaluating students’ class engagement. The proposed joint engagement evaluation model outperforms the single index model. The proposed weighted test scheme combining knowledge point test and self-test of comprehension degree validates the joint engagement degree model.
Key words： Deep Learning      Engagement Evaluation      Face Recognition     
Received: 14 May 2022      Published: 09 August 2023
ZTFLH:  TP393  
Fund:Key Project of Jilin Higher Education Teaching Reform Research(JLJY202269718747);National Social Science Fund of China(22BTQ048);Jilin Provincial Department of Education Social Science Project(JJKH20230195SK)
Corresponding Authors: Wang Qi，ORCID：0000-0003-1058-5068，E-mail：feelikesummer0108@163.com。   
Service
E-mail this article
Add to my bookshelf
Add to citation manager
E-mail Alert
RSS
Articles by authors
Nan Wang
Qi Wang

Cite this article:

Wang Nan, Wang Qi. Evaluating Student Engagement with Deep Learning. Data Analysis and Knowledge Discovery, 2023, 7(6): 123-133.

URL:

https://manu44.magtech.com.cn/Jwk_infotech_wk3/EN/10.11925/infotech.2096-3467.2022.0485     OR     https://manu44.magtech.com.cn/Jwk_infotech_wk3/EN/Y2023/V7/I6/123

Research Schema
Model Architecture
Accuracy of VGG16+Dense+Dropout （lr =1e-5）
模型架构 在测试集上
的准确率		 在验证集上
的准确率
VGG16（lr=1e-8） 0.78 0.83
VGG16+Dense+Dropout 0.80 0.85
VGG16+Dense 0.83 0.86
VGG16+Dense+Dropout（lr=1e-5） 0.93 0.92
The Accuracy of Different Model Architectures
面部表情 学生表情描述 权重（释义）
倦怠 下眼睑上扬，眉峰偏低，
嘴角向下		 -2（较为不满情绪的
表情）
困惑 微微咧嘴，眉头出现
皱纹上扬		 -1（稍微不满情绪的
表情）
表情不可见 无法看到面部器官及
神态		 0（与课堂环境无关的
表情）
无表情 整体肌肉松弛，瞳孔
正常放大		 1（稍微满意情绪的
表情，因其可以表示
学生在思考）
轻松 嘴唇上扬，眼睛眯起 2（较为满意情绪的
表情）
Student Emotion Weight in Class
专注水平 分数
低水平 [0，0.50）
中水平 [0.50，0.75）
高水平 [0.75，1）
Students’ Engagement Level in Class
Network Layers and Parameters
Loss Value of Training Network
Predicted Images
student21’s Expression Index Engagement Detection
student21’s Joint Index Engagement Detection
幻灯片结束的秒数 幻灯片对应知识点序号 幻灯片编号
18.72 1 1
50.85 1 2
93.82 2 3
183.96 3 4
281.96 4 5
309.77 5 6
387.52 5 7
418.90 6 8
449.84 6 9
615.46 7 10
822.78 7 11
852.20 8 12
1 018.70 8 13
1 063.50 8 14
1 091.80 8 15
1 199.70 8 16
The Correspondence Between Knowledge Points and Video Time
知识点序号 得分 知识点序号 得分
1 1 5 1
2 1 6 1
3 1 7 0.5
4 1 8 0.5
The Scores of Knowledge Points
图片文件名 图片所在视频时间点/s 对应知识点序号 对应幻灯片编号 知识点加权分 自测水平 测试总分
stu4660 37 1 2 0.368142 1 0.620885
stu4661 42 1 2 0.368142 1 0.620885
stu4662 47 1 2 0.368142 1 0.620885
stu4663 52 2 3 1 1 1
stu4664 57 2 3 1 1 1
stu4665 62 2 3 1 1 1
stu4666 67 2 3 1 1 1
stu4667 72 2 3 1 1 1
stu4668 77 2 3 1 1 1
stu4669 82 2 3 1 1 1
stu4670 87 2 3 1 1 1
stu4671 92 2 3 1 1 1
stu4672 97 3 4 0 0.6 0.24
… … … … … … …
student21’s Test Weighted Score （Partly）
student21’s Weighted Test Scores at Various Time Points
[1] Howard W R. Pattern Recognition and Machine Learning[J]. Kybernetes, 2007, 36(2): 275.
doi: 10.1108/03684920710743466
[2] LeCun Y, Bengio Y, Hinton G. Deep Learning[J]. Nature, 2015, 521(7553): 436-444.
doi: 10.1038/nature14539
[3] Coan J A, Allen J J B. Frontal EEG Asymmetry as a Moderator and Mediator of Emotion[J]. Biological Psychology, 2004, 67(1-2):7-50.
doi: 10.1016/j.biopsycho.2004.03.002 pmid: 15130524
[4] Day M. Exploiting Facial Landmarks for Emotion Recognition in the Wild[OL]. arXiv Preprint, arXiv: 1603.09129.
[5] LeCun Y, Bottou L, Bengio Y, et al. Gradient-Based Learning Applied to Document Recognition[J]. Proceedings of the IEEE, 1998, 86(11): 2278-2324.
doi: 10.1109/5.726791
[6] Dinesh D, S. A N, Bijlani K. Student Analytics for Productive Teaching/Learning[C]// Proceedings of International Conference on Information Science. 2017: 97-102.
[7] Butko N J, Theocharous G, Philipose M, et al. Automated Facial Affect Analysis for One-on-One Tutoring Applications[C]// Proceedings of 2011 IEEE International Conference on Automatic Face & Gesture Recognition. 2011: 382-387.
[8] Hedges J H, Adolph K E, Amso D, et al. Play, Attention, and Learning: How Do Play and Timing Shape the Development of Attention and Influence Classroom Learning?[J]. Annals of the New York Academy of Sciences, 2013, 1292(1): 1-20.
doi: 10.1111/nyas.2013.1292.issue-1
[9] Smith L B, Colunga E, Yoshida H. Knowledge as Process: Contextually Cued Attention and Early Word Learning[J]. Cognitive Science, 2010, 34(7): 1287-1314.
doi: 10.1111/j.1551-6709.2010.01130.x
[10] Steinmayr R, Ziegler M, Träuble B. Do Intelligence and Sustained Attention Interact in Predicting Academic Achievement?[J]. Learning and Individual Differences, 2010, 20(1): 14-18.
doi: 10.1016/j.lindif.2009.10.009
[11] Chen C M, Huang S H. Web-Based Reading Annotation System with an Attention-Based Self-Regulated Learning Mechanism for Promoting Reading Performance[J]. British Journal of Educational Technology, 2014, 45(5): 959-980.
doi: 10.1111/bjet.2014.45.issue-5
[12] 伍万杰. 基于姿态分析的学生学习状态监测方法研究与实现[D]. 重庆: 西南大学, 2021.
[12] (Wu Wanjie. Research and Implementation of Students’ Learning State Monitoring Method Based on Posture Analysis[D]. Chongqing: Southwest University, 2021.)
[13] Zaletelj J, Košir A. Predicting Students’ Attention in the Classroom from Kinect Facial and Body Features[J]. EURASIP Journal on Image and Video Processing, 2017, 2017: 80.
doi: 10.1186/s13640-017-0228-8
[14] 段巨力. 基于机器视觉的学生上课专注度的分析评测系统[D]. 杭州: 浙江工商大学, 2018.
[14] (Duan Juli. Evaluation and Evaluation System of Students’ Attentiveness Based on Machine Vision[D]. Hangzhou: Zhejiang Gongshang University, 2018.)
[15] 左国才, 王海东, 陈林书, 等. 基于人脸识别技术的现代学徒制学习效果评价研究[J]. 智能计算机与应用, 2019, 9(2): 116-118.
[15] (Zuo Guocai, Wang Haidong, Chen Linshu, et al. Evaluation of Modern Apprenticeship Learning Effect Based on Face Recognition Technology[J]. Intelligent Computer and Applications, 2019, 9(2): 116-118.)
[16] 钟马驰, 张俊朗, 蓝扬波, 等. 基于人脸检测和模糊综合评判的在线教育专注度研究[J]. 计算机科学, 2020, 47(S2): 196-203.
[16] (Zhong Machi, Zhang Junlang, Lan Yangbo, et al. Study on Online Education Focus Degree Based on Face Detection and Fuzzy Comprehensive Evaluation[J]. Computer Science, 2020, 47(S2): 196-203.)
[17] 何秀玲, 高倩, 李洋洋, 等. 基于深度学习模型的自发学习表情识别方法研究[J]. 计算机应用与软件, 2019, 36(3): 180-186.
[17] (He Xiuling, Gao Qian, Li Yangyang, et al. Spontaneous Learning Facial Expression Recognition Based on Deep Learning[J]. Computer Applications and Software, 2019, 36(3): 180-186.)
[18] Soloviev V. Machine Learning Approach for Student Engagement Automatic Recognition from Facial Expressions[J]. Scientific Publications of the State University of Novi Pazar Series A: Applied Mathematics Informatics and Mechanics, 2018, 10(2): 79-86.
doi: 10.5937/SPSUNP1802079S
[19] Mehrabian A. Communication Without Words[A]//Communication Theory[M]. London: Routledge, 2017: 193-200.
[20] Cowen A S, Keltner D. Self-Report Captures 27 Distinct Categories of Emotion Bridged by Continuous Gradients[J]. Proceedings of the National Academy of Sciences of the United States of America, 2017, 114(38): E7900-E7909.
[21] 郭晓旭. 基于微表情识别的学生课堂专注度分析系统研究[D]. 昆明: 云南师范大学, 2019.
[21] (Guo Xiaoxu. Research on the Analysis System of Students’ Classroom Attention Analysis Based on Micro-expression Recognition[D]. Kunming: Yunnan Normal University, 2019.)
[1] Wu Jialun, Zhang Ruonan, Kang Wulin, Yuan Puwei. Deep Learning Model of Drug Recommendation Based on Patient Similarity Analysis[J]. 数据分析与知识发现, 2023, 7(6): 148-160.
[2] Wang Xiaofeng, Sun Yujie, Wang Huazhen, Zhang Hengzhang. Construction and Verification of Type-Controllable Question Generation Model Based on Deep Learning and Knowledge Graphs[J]. 数据分析与知识发现, 2023, 7(6): 26-37.
[3] Liu Yang, Zhang Wen, Hu Yi, Mao Jin, Huang Fei. Hotel Stock Prediction Based on Multimodal Deep Learning[J]. 数据分析与知识发现, 2023, 7(5): 21-32.
[4] Huang Xuejian, Ma Tinghuai, Wang Gensheng. Detecting Weibo Rumors Based on Hierarchical Semantic Feature Learning Model[J]. 数据分析与知识发现, 2023, 7(5): 81-91.
[5] Wang Yinqiu, Yu Wei, Chen Junpeng. Automatic Question-Answering in Chinese Medical Q & A Community with Knowledge Graph[J]. 数据分析与知识发现, 2023, 7(3): 97-109.
[6] Zhang Zhengang, Yu Chuanming. Knowledge Graph Completion Model Based on Entity and Relation Fusion[J]. 数据分析与知识发现, 2023, 7(2): 15-25.
[7] Shen Lining, Yang Jiayi, Pei Jiaxuan, Cao Guang, Chen Gongzheng. A Fine-Grained Sentiment Recognition Method Based on OCC Model and Triggering Events[J]. 数据分析与知识发现, 2023, 7(2): 72-85.
[8] Wang Weijun, Ning Zhiyuan, Du Yi, Zhou Yuanchun. Identifying Interdisciplinary Sci-Tech Literature Based on Multi-Label Classification[J]. 数据分析与知识发现, 2023, 7(1): 102-112.
[9] Xiao Yuhan, Lin Huiping. Mining Differentiated Demands with Aspect Word Extraction: Case Study of Smartphone Reviews[J]. 数据分析与知识发现, 2023, 7(1): 63-75.
[10] Cheng Quan, She Dexin. Drug Recommendation Based on Graph Neural Network with Patient Signs and Medication Data[J]. 数据分析与知识发现, 2022, 6(9): 113-124.
[11] Wang Lu, Le Xiaoqiu. Research Progress on Citation Analysis of Scientific Papers[J]. 数据分析与知识发现, 2022, 6(4): 1-15.
[12] Zheng Xiao, Li Shuqing, Zhang Zhiwang. Measuring User Item Quality with Rating Analysis for Deep Recommendation Model[J]. 数据分析与知识发现, 2022, 6(4): 39-48.
[13] Yu Chuanming, Lin Hongjun, Zhang Zhengang. Joint Extraction Model for Entities and Events with Multi-task Deep Learning[J]. 数据分析与知识发现, 2022, 6(2/3): 117-128.
[14] Zhang Yunqiu, Li Bocheng, Chen Yan. Automatic Classification with Unbalanced Data for Electronic Medical Records[J]. 数据分析与知识发现, 2022, 6(2/3): 233-241.
[15] Zhang Fangcong, Qin Qiuli, Jiang Yong, Zhuang Runtao. Named Entity Recognition for Chinese EMR with RoBERTa-WWM-BiLSTM-CRF[J]. 数据分析与知识发现, 2022, 6(2/3): 251-262.
  Copyright © 2016 Data Analysis and Knowledge Discovery   Tel/Fax:(010)82626611-6626，82624938   E-mail:jishu@mail.las.ac.cn