Automated assessment and grading of computerized algorithms

• US 10,140,886 B2
• Filed: 03/23/2016
• Issued: 11/27/2018
• Est. Priority Date: 03/23/2016
• Status: Active Grant

• Current Assignee: Data Science Evangelists, Inc.
• Original Assignee: Data Science Evangelists, Inc.
• Inventors: Du, Alan, Mndange-Pfupfu, Ariel, Moscardi, Christian, Li, Tianhui
• Primary Examiner(s): McClellan, James S
• Application Number:
  US15/078,708
• Publication Number:
  US 20170278421A1
• Time in Patent Office: 979 Days
• CPC Class Codes: G06F 8/31 Programming languages or programming paradigms
• CPC Class Codes: G06F 8/41 Compilation
• CPC Class Codes: G06F 8/43 Checking; Contextual analysis
• CPC Class Codes: G06F 8/71 Version control security arrangements therefor G06F21/57; Configuration management
• CPC Class Codes: G06F 9/445 Program loading or initiating bootstrapping G06F9/4401; security arrangements for program loading or initiating G06F21/57
• CPC Class Codes: G06F 9/45504 Abstract machines for programme code execution, e.g. Java virtual machine [JVM], interpreters, emulators
• CPC Class Codes: G09B 19/0053 Computers, e.g. programming
• CPC Class Codes: G09B 7/02 of the type wherein the student is expected to construct an answer to the question which is presented or wherein the machine gives an answer to the question presented by a student

Abstract

A grading system and method for grading a user solution to a computing assignment are presented. The method includes receiving a program code submitted by a user, wherein the received program code is the user solution to the computing assignment; activating at least one code processing engine, wherein each code processing engine is a secured isolated execution environment; executing the program code in the at least one activated code processing engine to produce an answer; determining a grade for the answer based on an expected answer and an approximate grading function, wherein the approximate grading function is determined based on a type of the computing assignment; and returning the grade to the user.

Claims

1. A method for grading a user solution to a computing assignment, comprising:

receiving a program code submitted by a user, wherein the received program code is the user solution to the computing assignment;

activating a plurality of code execution engines, wherein each code execution engine of the plurality of code execution engines is a secured isolated execution environment;

executing the program code in the plurality of code execution engines concurrently to produce an answer;

determining a grade for the answer based on an expected answer and an approximate grading function, wherein the approximate grading function is determined based on a type of the computing assignment; and

returning the grade to the user.

2. The method of claim 1, further comprising:

authenticating the user prior to an assignment of the computing assignment to the user.

3. The method of claim 1, further comprising:

storing the received program code in a code repository with a unique version identifier; and

pushing the program code from the code repository to at least one code execution engine of the plurality of code execution engines.

4. The method of claim 1, further comprising:

inputting an input dataset to at least one code execution engine of the plurality of code execution engines, wherein the answer is produced in part based on the input dataset.

5. The method of claim 1, wherein executing the program code in the plurality of code execution engines further comprises:

checking the program code for at least syntactic correctness;

compiling, by the plurality of code execution engines, the program code;

determining whether at least one compilation error occurs during the compilation;

capturing the at least one compilation error, when it is determined that the at least one compilation error occurs during the compilation; and

reporting the captured at least one compilation error to the user.

6. The method of claim 1, further comprising:

validating the answer, wherein the grade is only determined for the answer when the answer is validated.

7. The method of claim 1, wherein the approximate grading function is any of:

a fuzzy logic function, a machine learning metric, and a time series metric.

8. The method of claim 7, wherein the fuzzy logic function includes at least one of:

a commonality matching function, a flat tail distribution function, and a forgiveness grading function.

9. The method of claim 7, wherein the machine learning metric includes at least one of:

a classification metric and a regression metric, wherein the regression metric includes any of;

a root mean square error (RMSE), a mean absolute error, and R2, wherein the classification metric includes any of;

a recall-and-precision, a F-score, an area under curve (AUC), a confusion score, and a jaccard similarity.

10. The method of claim 1, wherein the computing assignment is designed to train the user based on at least one of:

machine learning;

data wrangling;

extracting, transforming, and loading (ETL);

distributed computation and Hadoop;

distributed computing with Spark®

, Hive, and Scalding tools; and

data visualization.

11. The method of claim 1, wherein the approximate grading of a machine learning assignment further comprises:

predicting the expected answer based on the answer and a test validated dataset, wherein the test validated dataset is randomly selected from a group of test validated datasets.

12. The method of claim 1, wherein the program code is written using any of:

a programming language, and a scripting language.

13. The method of claim 1, wherein each code execution engine is any of:

a virtual container, and a software container.

14. A non-transitory computer readable medium having stored thereon instructions for causing one or more processing units to execute the method according to claim 1.

15. A grading system for grading a user solution to a computing assignment, comprising:

a processing system; and

a memory, the memory containing instructions that, when executed by the processing system, configure the processing system to;

receive a program code submitted by a user, wherein the received program code is the user solution to the computing assignment;

activate a plurality of code execution engines, wherein each code execution engine of the plurality of code execution engines is a secured isolated execution environment;

execute the program code in the plurality of code execution engines concurrently to produce an answer;

determine a grade for the answer based on an expected answer and an approximate grading function, wherein the approximate grading function is determined based on a type of the computing assignment; and

return the grade to the user.

16. The grading system of claim 15, wherein the grading system is further configured to:

authenticate the user prior to an assignment of the computing assignment to the user.

17. The grading system of claim 15, wherein the grading system is further configured to:

store the received program code in a code repository with a unique version identifier; and

push the program code from the code repository to at least one code execution engine of the plurality of code execution engines.

18. The grading system of claim 15, wherein the grading system is further configured to:

input an input dataset to at least one code execution engine of the plurality of code execution engines, wherein the answer is produced in part based on the input dataset.

19. The grading system of claim 15, wherein the grading system is further configured to:

check the program code for at least syntactic correctness;

compile, by the plurality of code execution engines, the program code;

determine whether at least one compilation error occurs during the compilation;

capture the at least one compilation error, when it is determined that the at least one compilation error occurs during the compilation; and

report the captured at least one compilation error to the user.

20. The grading system of claim 15, wherein the grading system is further configured to:

validate the answer, wherein the grade is only determined for the answer when the answer is validated.

21. The grading system of claim 15, wherein the approximate grading function is any of:

a fuzzy logic function, a machine learning metric, and a time series metric.

22. The grading system of claim 21, wherein the fuzzy logic function includes at least one of:

a commonality matching function, a flat tail distribution function, and a forgiveness grading function.

23. The grading system of claim 22, wherein the machine learning metric includes at least one of:

a classification metric and a regression metric, wherein the regression metric includes any of;

a root mean square error (RMSE), a mean absolute error, and R2, wherein the classification metric includes any of;

a recall-and-precision, a F-score, an area under curve (AUC), a confusion score, and a jaccard similarity.

24. The grading system of claim 15, wherein the computing assignment is designed to train the user based on at least one of:

machine learning;

data wrangling;

extracting, transforming, and loading (ETL);

distributed computation and Hadoop;

distributed computing with Spark®

, HIVE, and Scalding tools; and

data visualization.

25. The grading system of claim 15, wherein the grading system is further configured to approximate grading of a machine learning assignment by predicting the expected answer using the answer and a test validated dataset, wherein the test validated dataset is randomly selected from a group of test validated datasets.

26. The grading system claim 15, wherein the program code is written using any of:

a programming language, and a scripting language.

27. The grading system of claim 15, wherein each code execution engine is any of:

a virtual container, and a software container.