[Objective] This paper builds an automatic scoring system for subjective questions in the maritime competency assessment, aiming to reduce the heavy workload and human factors of subjective question scoring. [Methods] Firstly, we used the weighted TextRank algorithm of dependency syntax analysis to extract keywords. Then, we integrated sentence vectors, core words, syntactic components, and dependent structures to judge the similarity between student answers and the standard ones. Third, we constructed a set of special negative words for maritime affairs to judge the semantic opposition between the student’s answer and the standard answer. Finally, we gave each answer an objective score. [Results] We examined our method with multiple sets of different subjective questions, and found the average score difference between the automatic score and the manual scoring was 0.21, with a deviation rate of 4.20%. [Limitations] More research is needed to improve the processing of long and complex sentences. [Conclusions] The proposed algorithm could effectively evaluate subjective questions in the maritime competency assessment.
韩辉, 刘秀文. 海事适任评估中主观题自动评分技术研究*[J]. 数据分析与知识发现, 2021, 5(8): 113-121.
Han Hui, Liu Xiuwen. Automatic Scoring for Subjective Questions in Maritime Competency Assessment. Data Analysis and Knowledge Discovery, 2021, 5(8): 113-121.
( China Maritime Safety Administration. Guidelines for the Implementation of the Manila Amendment to the STCW Convention[M]. Dalian: Dalian Maritime University Press, 2010.)
[2]
Page E B. Project Essay Grade: PEG[A]//Shermis M D, Burstein J. Automated Essay Scoring: A Cross-disciplinary Perspective[M]. Lawrence Erlbaum Associates Publishers, 2003: 43-54.
[3]
Foltz P W, Streeter L E, Lochbaum K E, et al. Implementation and Applications of the Intelligent Essay Assessor[A]//Shermis M D, Burstein J. Handbook of Automated Essay Evaluation[M]2013:68-88.
[4]
Burstein J, Tetreault J, Madnani N. The E-rater Automated Essay Scoring System[A]//Handbook of Automated Essay Evaluation: Current Applications and New Directions[M]. New York: Routledge, 2013: 55-67.
[5]
Burrows S, Gurevych I, Stein B. The Eras and Trends of Automatic Short Answer Grading[J]. International Journal of Artificial Intelligence in Education, 2015, 25:60-117.
[6]
Chapelle C A, Chung Y R. The Promise of NLP and Speech Processing Technologies in Language Assessment[J]. Language Testing, 2010, 27(3):301-315.
doi: 10.1177/0265532210364405
[7]
Vij S, Tayal D, Jain A. A Machine Learning Approach for Automated Evaluation of Short Answers Using Text Similarity Based on WordNet Graphs[J]. Wireless Personal Communications, 2020, 111(2):1271-1282.
doi: 10.1007/s11277-019-06913-x
( Gao Sidan. Research and Preliminary Realization of Subjective Examination Questions Automatic Correction Technology Based on Natural Language Understanding[D]. Nanjing: Nanjing University, 2003.)
( Meng Aiguo, Bu Shengxian, Li Ying, et al. Design and Implementation of an Automated Assessment Algorithm for Subjective Tests in Network Examination System[J]. Computer and Digital Engineering, 2005, 33(7):147-150.)
( Li Huiyang, Han Zhongyuan. Auto-Judging to the Simple Description in the Specific Field and Its Application in CAI[J]. Computer Engineering and Applications, 2002, 38(8):76-78.)
( Li Xuejun. Algorithm Realization of the Subjective Question’s Auto-Marking Based on the Artificial Intelligence[J]. Journal of Jiangnan University (Natural Science Edition), 2009, 8(3):292-295.)
[12]
张胜楠. 基于文本相似度的短文本主观题自动评分方法研究[D]. 武汉: 武汉理工大学, 2017.
[12]
( Zhang Shengnan. Research on Automatic Scoring Method of Short Text Subjective Questions Based on Text Similarity[D]. Wuhan: Wuhan University of Technology, 2017.)
[13]
代钰琴. 智能评卷技术的应用研究[D]. 绵阳: 西南科技大学, 2019.
[13]
( Dai Yuqin. Research on the Application of Intelligent Marking Technology[D]. Mianyang: Southwest University of Science and Technology, 2019.)
[14]
Zhang W N, Ming Z Y, Zhang Y, et al. The Use of Dependency Relation Graph to Enhance the Term Weighting in Question Retrieval[C]// Proceedings of COLING 2012: Technical Papers. 2012: 3105-3120.
[15]
Mihalcea R, Tarau P. TextRank: Bringing Order into Texts[C]// Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing. 2004: 404-411.
( Xia Tian. Extracting Keywords with Modified TextRank Model[J]. Data Analysis and Knowledge Discovery, 2017, 1(2):28-34.)
[17]
Mikolov T, Sutskever L, Chen K, et al. Distributed Representations of Words and Phrases and Their Compositionality[C]// Proceedings of the 26th International Conference on Neural Information Processing Systems. 2013: 3111-3119.
[18]
Devlin J, Chang M W, Lee K, et al. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding[C]// Proceeding of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2019: 4171-4186.
( Mai Fanjin, Wang Ting. A Computational Model of the Contrary Degree Between Two Words[J]. Journal of Chinese Information Processing, 2008, 22(4):39-42.)
( Liang Xiaocheng, Yue Xiaoguang, Mai Fanjin, et al. Algorithm Research of Subjective Question Assessment Based on Natural Language Processing[J]. Journal of Kunming University of Science and Technology (Science and Technology Edition), 2010, 35(2):81-84.)
[21]
Peng B, Wang J, Zhang X J. Adversarial Learning of Sentiment Word Representations for Sentiment Analysis[J]. Information Sciences, 2020, 541:426-441.
doi: 10.1016/j.ins.2020.06.044
[22]
Al-Khalifa H S, Al-Yahya M M, Bahanshal A, et al. SemQ: A Proposed Framework for Representing Semantic Opposition in the Holy Quran Using Semantic Web Technologies[C]// Proceedings of 2009 International Conference on the Current Trends in Information Technology. DOI: 10.1109/CTIT.2009.5423145.
doi: 10.1109/CTIT.2009.5423145
