Please wait a minute...
Data Analysis and Knowledge Discovery  2021, Vol. 5 Issue (8): 113-121    DOI: 10.11925/infotech.2096-3467.2020.1193
Current Issue | Archive | Adv Search |
Automatic Scoring for Subjective Questions in Maritime Competency Assessment
Han Hui(),Liu Xiuwen
Navigation College, Dalian Maritime University, Dalian 116000, China
Download: PDF (1054 KB)   HTML ( 7
Export: BibTeX | EndNote (RIS)      
Abstract  

[Objective] This paper builds an automatic scoring system for subjective questions in the maritime competency assessment, aiming to reduce the heavy workload and human factors of subjective question scoring. [Methods] Firstly, we used the weighted TextRank algorithm of dependency syntax analysis to extract keywords. Then, we integrated sentence vectors, core words, syntactic components, and dependent structures to judge the similarity between student answers and the standard ones. Third, we constructed a set of special negative words for maritime affairs to judge the semantic opposition between the student’s answer and the standard answer. Finally, we gave each answer an objective score. [Results] We examined our method with multiple sets of different subjective questions, and found the average score difference between the automatic score and the manual scoring was 0.21, with a deviation rate of 4.20%. [Limitations] More research is needed to improve the processing of long and complex sentences. [Conclusions] The proposed algorithm could effectively evaluate subjective questions in the maritime competency assessment.

Key wordsAutomatic Scoring of Subjective Questions      Extraction      Similarity Calculation      Opposition Judgment      Maritime Field     
Received: 30 November 2020      Published: 15 September 2021
ZTFLH:  TP391  
Fund:Ministry of Industry and Information Technology([2018]No.473);Fundamental Research Funds for the Central Universities(3132019312)
Corresponding Authors: Han Hui ORCID:0000-0002-3053-0614     E-mail: hanh1201@126.com

Cite this article:

Han Hui, Liu Xiuwen. Automatic Scoring for Subjective Questions in Maritime Competency Assessment. Data Analysis and Knowledge Discovery, 2021, 5(8): 113-121.

URL:

https://manu44.magtech.com.cn/Jwk_infotech_wk3/EN/10.11925/infotech.2096-3467.2020.1193     OR     https://manu44.magtech.com.cn/Jwk_infotech_wk3/EN/Y2021/V5/I8/113

Flow Chart of Automatic Scoring Algorithm
词性 概率分布
nz 40.67%
v 23.46%
n 20.21%
m 12.20%
b 3.46%
Keyword Part of Speech Distribution
Keyword Extraction Flowchart
文本切分
结果
说明 对立度判断
A类 标准答案与学生答案均不可切分 直接计算标准答案与学生答案中否定词的数量差
B类 标准答案不可切分,学生答案可切分为多句 计算学生答案所有分句中否定词个数之和与标准答案中否定词的数量差
C类 标准答案可切分为多句,学生答案不可切分 计算学生答案中否定词与标准答案第一分句中否定词的数量差
D类 标准答案与学生答案均可切分为多句 先计算学生答案第一分句中否定词与标准答案第一分句中否定词的数量差,然后计算学生答案和标准答案其余分句否定词个数和的差
Classification of the Degree of Opposition
文本切分结果 说明 评分公式
A,B,C类 学生答案与标准答案至少其一不可切分 M = 5 × ( ω × Co n Keywords + ( 1 - ω ) × Si m sen ) × O
D类 学生答案与标准答案第一分句对立,其余分句不对立 M = 2.5 × ( ω × Co n Keywords + ( 1 - ω ) × Si m sen )
学生答案与标准答案第一分句对立,其余分句也对立 M = 0
学生答案与标准答案第一分句不对立,其余分句对立 M = 2.5 × ( ω × Co n Keywords + ( 1 - ω ) × Si m sen )
学生答案与标准答案第一分句不对立,其余分句也不对立 M = 5 × ( ω × Co n Keywords + ( 1 - ω ) × Si m sen )
Scoring Formula in Different Situations
The Average Score Difference of the System Score under Different Values of ω
Distribution of Similarity Between System and Manual Scoring
项目 标准答案 学生答案
答案内容 潮汐主要是由天体引潮力作为原动力,由于其距离的不同形成了潮汐椭圆体,再结合月球公转以及地球自转、公转,其相对位置不断变换,造成了潮汐一天之内有高低潮现象。 潮汐主要是由天体引潮力作为原动力,海水随地球自转的同时,受到了月球和太阳的引力,在日,月引潮力的作用下引起的海面周期性的升降,称为海洋潮汐。
关键词 ‘天体引潮力’,‘高低潮’,‘潮汐椭球体’,‘位置’ ‘天体引潮力’,‘引力’,‘升降’
Example Analysis of Manual Scoring and System Scoring
[1] 中华人民共和国海事局. STCW公约马尼拉修正案履约指南[M]. 大连: 大连海事大学出版社, 2010.
[1] ( China Maritime Safety Administration. Guidelines for the Implementation of the Manila Amendment to the STCW Convention[M]. Dalian: Dalian Maritime University Press, 2010.)
[2] Page E B. Project Essay Grade: PEG[A]//Shermis M D, Burstein J. Automated Essay Scoring: A Cross-disciplinary Perspective[M]. Lawrence Erlbaum Associates Publishers, 2003: 43-54.
[3] Foltz P W, Streeter L E, Lochbaum K E, et al. Implementation and Applications of the Intelligent Essay Assessor[A]//Shermis M D, Burstein J. Handbook of Automated Essay Evaluation[M]2013:68-88.
[4] Burstein J, Tetreault J, Madnani N. The E-rater Automated Essay Scoring System[A]//Handbook of Automated Essay Evaluation: Current Applications and New Directions[M]. New York: Routledge, 2013: 55-67.
[5] Burrows S, Gurevych I, Stein B. The Eras and Trends of Automatic Short Answer Grading[J]. International Journal of Artificial Intelligence in Education, 2015, 25:60-117.
[6] Chapelle C A, Chung Y R. The Promise of NLP and Speech Processing Technologies in Language Assessment[J]. Language Testing, 2010, 27(3):301-315.
doi: 10.1177/0265532210364405
[7] Vij S, Tayal D, Jain A. A Machine Learning Approach for Automated Evaluation of Short Answers Using Text Similarity Based on WordNet Graphs[J]. Wireless Personal Communications, 2020, 111(2):1271-1282.
doi: 10.1007/s11277-019-06913-x
[8] 高思丹. 基于自然语言理解的主观试题自动批改技术的研究与初步实现[D]. 南京: 南京大学, 2003.
[8] ( Gao Sidan. Research and Preliminary Realization of Subjective Examination Questions Automatic Correction Technology Based on Natural Language Understanding[D]. Nanjing: Nanjing University, 2003.)
[9] 孟爱国, 卜胜贤, 李鹰, 等. 一种网络考试系统中主观题自动评分的算法设计与实现[J]. 计算机与数字工程, 2005, 33(7):147-150.
[9] ( Meng Aiguo, Bu Shengxian, Li Ying, et al. Design and Implementation of an Automated Assessment Algorithm for Subjective Tests in Network Examination System[J]. Computer and Digital Engineering, 2005, 33(7):147-150.)
[10] 李辉阳, 韩忠愿. 有限领域简述文字的自动判读及其在CAI中的应用[J]. 计算机工程与应用, 2002, 38(8):76-78.
[10] ( Li Huiyang, Han Zhongyuan. Auto-Judging to the Simple Description in the Specific Field and Its Application in CAI[J]. Computer Engineering and Applications, 2002, 38(8):76-78.)
[11] 李学俊. 基于人工智能的主观题自动评分算法实现[J]. 江南大学学报(自然科学版), 2009, 8(3):292-295.
[11] ( Li Xuejun. Algorithm Realization of the Subjective Question’s Auto-Marking Based on the Artificial Intelligence[J]. Journal of Jiangnan University (Natural Science Edition), 2009, 8(3):292-295.)
[12] 张胜楠. 基于文本相似度的短文本主观题自动评分方法研究[D]. 武汉: 武汉理工大学, 2017.
[12] ( Zhang Shengnan. Research on Automatic Scoring Method of Short Text Subjective Questions Based on Text Similarity[D]. Wuhan: Wuhan University of Technology, 2017.)
[13] 代钰琴. 智能评卷技术的应用研究[D]. 绵阳: 西南科技大学, 2019.
[13] ( Dai Yuqin. Research on the Application of Intelligent Marking Technology[D]. Mianyang: Southwest University of Science and Technology, 2019.)
[14] Zhang W N, Ming Z Y, Zhang Y, et al. The Use of Dependency Relation Graph to Enhance the Term Weighting in Question Retrieval[C]// Proceedings of COLING 2012: Technical Papers. 2012: 3105-3120.
[15] Mihalcea R, Tarau P. TextRank: Bringing Order into Texts[C]// Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing. 2004: 404-411.
[16] 夏天. 词向量聚类加权TextRank的关键词抽取[J]. 数据分析与知识发现, 2017, 1(2):28-34.
[16] ( Xia Tian. Extracting Keywords with Modified TextRank Model[J]. Data Analysis and Knowledge Discovery, 2017, 1(2):28-34.)
[17] Mikolov T, Sutskever L, Chen K, et al. Distributed Representations of Words and Phrases and Their Compositionality[C]// Proceedings of the 26th International Conference on Neural Information Processing Systems. 2013: 3111-3119.
[18] Devlin J, Chang M W, Lee K, et al. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding[C]// Proceeding of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2019: 4171-4186.
[19] 麦范金, 王挺. 语义对立度及其计算模型的研究[J]. 中文信息学报, 2008, 22(4):39-42.
[19] ( Mai Fanjin, Wang Ting. A Computational Model of the Contrary Degree Between Two Words[J]. Journal of Chinese Information Processing, 2008, 22(4):39-42.)
[20] 梁晓诚, 岳晓光, 麦范金, 等. 基于自然语言处理的主观题评分算法研究[J]. 昆明理工大学学报(理工版), 2010, 35(2):81-84.
[20] ( Liang Xiaocheng, Yue Xiaoguang, Mai Fanjin, et al. Algorithm Research of Subjective Question Assessment Based on Natural Language Processing[J]. Journal of Kunming University of Science and Technology (Science and Technology Edition), 2010, 35(2):81-84.)
[21] Peng B, Wang J, Zhang X J. Adversarial Learning of Sentiment Word Representations for Sentiment Analysis[J]. Information Sciences, 2020, 541:426-441.
doi: 10.1016/j.ins.2020.06.044
[22] Al-Khalifa H S, Al-Yahya M M, Bahanshal A, et al. SemQ: A Proposed Framework for Representing Semantic Opposition in the Holy Quran Using Semantic Web Technologies[C]// Proceedings of 2009 International Conference on the Current Trends in Information Technology. DOI: 10.1109/CTIT.2009.5423145.
doi: 10.1109/CTIT.2009.5423145
[1] Wang Yifan,Li Bo,Shi Hua,Miao Wei,Jiang Bin. Annotation Method for Extracting Entity Relationship from Ancient Chinese Works[J]. 数据分析与知识发现, 2021, 5(9): 63-74.
[2] Ma Jiangwei, Lv Xueqiang, You Xindong, Xiao Gang, Han Junmei. Extracting Relationship Among Military Domains with BERT and Relation Position Features[J]. 数据分析与知识发现, 2021, 5(8): 1-12.
[3] Chai Qingfeng, Shi Linyan, Mei Shan, Xiong Haitao, He Huixin. Extracting Knowledge Elements of Sci-Tech Literature Based on Artificial and Machine Features[J]. 数据分析与知识发现, 2021, 5(8): 132-144.
[4] Tan Ying, Tang Yifei. Extracting Citation Contents with Coreference Resolution[J]. 数据分析与知识发现, 2021, 5(8): 25-33.
[5] Zhang Jiandong, Chen Shiji, Xu Xiaoting, Zuo Wenge. Extracting PDF Tables Based on Word Vectors[J]. 数据分析与知识发现, 2021, 5(8): 34-44.
[6] Yu Xuehan, He Lin, Xu Jian. Extracting Events from Ancient Books Based on RoBERTa-CRF[J]. 数据分析与知识发现, 2021, 5(7): 26-35.
[7] Zhao Danning,Mu Dongmei,Bai Sen. Automatically Extracting Structural Elements of Sci-Tech Literature Abstracts Based on Deep Learning[J]. 数据分析与知识发现, 2021, 5(7): 70-80.
[8] Chen Xingyue, Ni Liping, Ni Zhiwei. Extracting Financial Events with ELECTRA and Part-of-Speech[J]. 数据分析与知识发现, 2021, 5(7): 36-47.
[9] Yan Qiang,Zhang Xiaoyan,Zhou Simin. Extracting Keywords Based on Sememe Similarity[J]. 数据分析与知识发现, 2021, 5(4): 80-89.
[10] Shi Xiang,Liu Ping. Extraction and Representation of Domain Knowledge with Semantic Description Model and Knowledge Elements——Case Study of Information Retrieval[J]. 数据分析与知识发现, 2021, 5(4): 123-133.
[11] Cheng Bin,Shi Shuicai,Du Yuncheng,Xiao Shibin. Keyword Extraction for Journals Based on Part-of-Speech and BiLSTM-CRF Combined Model[J]. 数据分析与知识发现, 2021, 5(3): 101-108.
[12] Hu Shaohu,Zhang Yingyi,Zhang Chengzhi. Review of Keyword Extraction Studies[J]. 数据分析与知识发现, 2021, 5(3): 45-59.
[13] Wu Yanwen, Cai Qiuting, Liu Zhi, Deng Yunze. Digital Resource Recommendation Based on Multi-Source Data and Scene Similarity Calculation[J]. 数据分析与知识发现, 2021, 5(11): 114-123.
[14] Zheng Xinman, Dong Yu. Constructing Degree Lexicon for STI Policy Texts[J]. 数据分析与知识发现, 2021, 5(10): 81-93.
[15] Dai Zhihong, Hao Xiaoling. Extracting Hypernym-Hyponym Relationship for Financial Market Applications[J]. 数据分析与知识发现, 2021, 5(10): 60-70.
  Copyright © 2016 Data Analysis and Knowledge Discovery   Tel/Fax:(010)82626611-6626,82624938   E-mail:jishu@mail.las.ac.cn