[Objective] This paper builds an automatic scoring system for subjective questions in the maritime competency assessment, aiming to reduce the heavy workload and human factors of subjective question scoring. [Methods] Firstly, we used the weighted TextRank algorithm of dependency syntax analysis to extract keywords. Then, we integrated sentence vectors, core words, syntactic components, and dependent structures to judge the similarity between student answers and the standard ones. Third, we constructed a set of special negative words for maritime affairs to judge the semantic opposition between the student’s answer and the standard answer. Finally, we gave each answer an objective score. [Results] We examined our method with multiple sets of different subjective questions, and found the average score difference between the automatic score and the manual scoring was 0.21, with a deviation rate of 4.20%. [Limitations] More research is needed to improve the processing of long and complex sentences. [Conclusions] The proposed algorithm could effectively evaluate subjective questions in the maritime competency assessment.
韩辉, 刘秀文. 海事适任评估中主观题自动评分技术研究*[J]. 数据分析与知识发现, 2021, 5(8): 113-121.
Han Hui, Liu Xiuwen. Automatic Scoring for Subjective Questions in Maritime Competency Assessment. Data Analysis and Knowledge Discovery, 2021, 5(8): 113-121.
( China Maritime Safety Administration. Guidelines for the Implementation of the Manila Amendment to the STCW Convention[M]. Dalian: Dalian Maritime University Press, 2010.)
Page E B. Project Essay Grade: PEG[A]//Shermis M D, Burstein J. Automated Essay Scoring: A Cross-disciplinary Perspective[M]. Lawrence Erlbaum Associates Publishers, 2003: 43-54.
Foltz P W, Streeter L E, Lochbaum K E, et al. Implementation and Applications of the Intelligent Essay Assessor[A]//Shermis M D, Burstein J. Handbook of Automated Essay Evaluation[M]2013:68-88.
Burstein J, Tetreault J, Madnani N. The E-rater Automated Essay Scoring System[A]//Handbook of Automated Essay Evaluation: Current Applications and New Directions[M]. New York: Routledge, 2013: 55-67.
Burrows S, Gurevych I, Stein B. The Eras and Trends of Automatic Short Answer Grading[J]. International Journal of Artificial Intelligence in Education, 2015, 25:60-117.
Chapelle C A, Chung Y R. The Promise of NLP and Speech Processing Technologies in Language Assessment[J]. Language Testing, 2010, 27(3):301-315.
Vij S, Tayal D, Jain A. A Machine Learning Approach for Automated Evaluation of Short Answers Using Text Similarity Based on WordNet Graphs[J]. Wireless Personal Communications, 2020, 111(2):1271-1282.
( Meng Aiguo, Bu Shengxian, Li Ying, et al. Design and Implementation of an Automated Assessment Algorithm for Subjective Tests in Network Examination System[J]. Computer and Digital Engineering, 2005, 33(7):147-150.)
( Xia Tian. Extracting Keywords with Modified TextRank Model[J]. Data Analysis and Knowledge Discovery, 2017, 1(2):28-34.)
Mikolov T, Sutskever L, Chen K, et al. Distributed Representations of Words and Phrases and Their Compositionality[C]// Proceedings of the 26th International Conference on Neural Information Processing Systems. 2013: 3111-3119.
Devlin J, Chang M W, Lee K, et al. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding[C]// Proceeding of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2019: 4171-4186.
( Liang Xiaocheng, Yue Xiaoguang, Mai Fanjin, et al. Algorithm Research of Subjective Question Assessment Based on Natural Language Processing[J]. Journal of Kunming University of Science and Technology (Science and Technology Edition), 2010, 35(2):81-84.)
Peng B, Wang J, Zhang X J. Adversarial Learning of Sentiment Word Representations for Sentiment Analysis[J]. Information Sciences, 2020, 541:426-441.
Al-Khalifa H S, Al-Yahya M M, Bahanshal A, et al. SemQ: A Proposed Framework for Representing Semantic Opposition in the Holy Quran Using Semantic Web Technologies[C]// Proceedings of 2009 International Conference on the Current Trends in Information Technology. DOI: 10.1109/CTIT.2009.5423145.