Automatic assessment via intelligent analysis of students’ program output patterns

Abstract

Automatic assessment of computer programming exercises offers a number of benefits to both learners and educators, including timely and customised feedback, as well as saving of human effort in grading. However, due to the high variety of programs submitted by students, exact matching between the expected output and different output variants is undesirable and how to do the matching properly is a challenging and practical problem. Existing approaches to address this problem adopt various inexact matching strategies, but typically they are unscalable, incapable of expressing a diversity of program outputs, or require high level of expertise. In this paper, we propose Hierarchical Program Output Structure (HiPOS), which provides higher expressiveness and flexibility, to model the program output. Based on HiPOS, we design different levels of matching rules in the matching rule hierarchy to determine the admissible program output variants in a flexible and scalable manner. We conducted experiments and compare our approach of automatic assessment to human judgement. The results show that our proposed method achieved an accuracy of 0.8467 in determining the admissible program output variants.