AUTOMATED EVALUATION OF PROGRAMMING ASSIGNMENTS

Abstract: Grading programming assignments by hand can be laborious and prone to mistakes. Feedback with failing test cases is generated using current tools. Nevertheless, this approach is ineffective and yields insufficient outcomes. In this paper, we provide AUTOGRADER, a tool that, given a single reference implementation of the issue, automatically assesses whether programming assignments are right and generates counterexamples. Our tool looks for semantically distinct execution pathways between a student's submission and the reference implementation rather than counting the passed tests. The proposal is considered wrong if such a discrepancy is discovered; if not, it is considered to be the right answer. To capture the semantics of execution paths and identify possible path variations, we employ symbolic execution and weakest preconditions. AUTOGRADER is the first automated grading tool that relies on program semantics and generates feedback with counterexamples based on path deviations. It also reduces human efforts in writing test cases and makes the grading more complete. We implement AUTOGRADER and test its effectiveness and performance with real-world programming problems and student submissions collected from an online programming site. Our experiment reveals that there are no false negatives using our proposed method and we detected 11 errors of online platform judges.

Keywords: Automated Assessment, Programming Assignments, Software Testing, Automated Grading, Evaluation of code