Automated evaluation of computer programming

• US 10,353,703 B1
• Filed: 01/27/2017
• Issued: 07/16/2019
• Est. Priority Date: 01/28/2016
• Status: Active Grant

• Current Assignee: Codesignal Incorporated
• Original Assignee: BrainFights, Inc.
• Inventors: Sloyan, Tigran, Shatakhtsyan, Aram, Desroches, Jan-Felix, Piliposyan, Eduard
• Primary Examiner(s): Tecklu, Isaac T
• Application Number:
  US15/418,342
• Time in Patent Office: 900 Days
• Field of Search: None
• CPC Class Codes: G06F 11/3616 using software metrics
• CPC Class Codes: G06F 11/3672 Test management
• CPC Class Codes: G06F 11/3688 for test execution, e.g. scheduling of test suites
• CPC Class Codes: G06F 11/3692 for test results analysis
• CPC Class Codes: G06F 8/30 Creation or generation of source code
• CPC Class Codes: G06F 8/71 Version control security arrangements therefor G06F21/57; Configuration management
• CPC Class Codes: G06F 8/77 Software metrics
• CPC Class Codes: G06F 9/505 considering the load

Abstract

Techniques are described for analysis and evaluation of different computer programs developed by different users to accomplish programming tasks. A computer programming evaluation system provides users with one or more programming tasks and ability to input source code to solve the programming tasks. In an embodiment, using a client computer system, a user initiates a computer programming session with a programming system. As a result of the session, the client computer system is provided with user interface elements allowing the user to view one or more programming tasks and submit to the programming system source code for the one or more programming tasks. In an embodiment, different modes are used by the programming system to evaluate quality of the computer programming performed by a user in terms of accomplishing the programming task. By collecting result metrics from evaluations and comparing with complexity data of the programming tasks, the programming system generates a graph-based quantified knowledge map of computer programming for a user. Non-limiting examples of modes for evaluation of the computer programming include a practice mode, an arcade mode, versus mode, a bot mode and a bot training mode. In an embodiment, to execute source codes in parallel, a programming system may include multiple programming servers. Each programming server is a separate computer system that, in some embodiments, is a cloud computer system allocated from cloud services. Executing different instances of source code on different programming servers improves both the security and scalability of the programming system.

Claims

1. A computer-implemented method comprising:

receiving one or more source codes submitted for a programming task in a programming session with a client computer system of a particular user;

executing the one or more source codes to generate result data for the programming task;

evaluating the result data using one or more metrics of the programming task;

based on complexity data of the programming task and based on evaluating the result data using the one or more metrics, determining whether to update a programming rating for the particular user;

wherein the programming rating for the particular user is associated with a graph of nodes that comprises a plurality of nodes and a plurality of edges, wherein each node in the plurality of nodes represents a respective programming subject and comprises a subject complexity rating for the particular user for the respective programming subject, said each node is connected to one or more neighbor nodes using one or more edges of the plurality of edges, wherein the one or more neighbor nodes represent one or more related programming subjects to the respective programming subject;

causing a display of the result data for the particular user on the client computer system of the particular user.

2. The method of claim 1, wherein the programming rating for the particular user includes a range of values.

3. The method of claim 1, wherein the complexity data comprises of a subject complexity rating for the programming task.

4. The method of claim 1, wherein the programming task is associated with one or more programming subjects, and the complexity data of the programming task comprises one or more respective subject complexity ratings for the one or more programming subjects, and the method further comprises:

determining one or more nodes of the graph of nodes for the particular user that correspond to the one or more programming subjects; and

based on the one or more respective subject complexity ratings of the complexity data of the programming task, updating one or more corresponding subject complexity ratings of the one or more nodes of the graph of nodes for the particular user.

5. The method of claim 4, further comprising, before the updating the one or more corresponding subject complexity ratings of the one or more nodes of the graph of nodes for the particular user, based on the one or more respective subject complexity ratings of the complexity data of the programming task, adjusting the one or more respective subject complexity ratings of the complexity data according to the result data.

6. The method of claim 4, further comprising, before the updating the one or more corresponding subject complexity ratings of the one or more nodes of the graph of nodes for the particular user, based on the one or more respective subject complexity ratings of the complexity data of the programming task, positively adjusting the one or more respective subject complexity ratings of the complexity data, when the result data indicates a shorter programming duration spent on the programming task by the particular user than an average programming duration for the programming task.

7. The method of claim 1, wherein the programming task is associated with one or more programming subjects and the complexity data of the programming task comprises one or more respective subject complexity ratings for the one or more programming subjects, and the method further comprises:

determining one or more nodes of the graph of nodes for the particular user that correspond to the one or more programming subjects;

determining one or more particular neighbor nodes of the one or more nodes by traversing one or more particular edges from the one or more nodes of the graph of nodes; and

based on the one or more respective subject complexity ratings of the complexity data of the programming task, updating one or more subject complexity ratings of the one or more particular neighbor nodes of the one or more nodes of the graph of nodes for the particular user.

8. The method of claim 1, wherein the programming task is associated with one or more programming subjects and the complexity data of the programming task comprises one or more respective subject complexity ratings for the one or more programming subjects, and the method further comprises:

determining one or more nodes of the graph of nodes for the particular user that correspond to the one or more programming subjects;

determining one or more particular neighbor nodes of the one or more nodes by traversing one or more particular edges from the one or more nodes of the graph of nodes; and

based on the one or more respective subject complexity ratings of the complexity data of the programming task, updating one or more subject complexity ratings of the one or more particular neighbor nodes of the one or more nodes of the graph of nodes for the particular user.

9. The method of claim 1, further comprising determining the programming rating for the particular user by aggregating a plurality of subject complexity ratings of the plurality of nodes of the graph of nodes for the particular user using one or more aggregation functions.

10. A server computer system comprising:

one or more processors;

one or more storage media storing one or more computer programs for execution by the one or more processors, the one or more computer programs comprising instructions for;

receiving one or more source codes submitted for a programming task in a programming session with a client computer system of a particular user;

executing the one or more source codes to generate result data for the programming task;

evaluating the result data using one or more metrics of the programming task;

based on complexity data of the programming task and based on evaluating the result data using the one or more metrics, determining whether to update a programming rating for the particular user;

wherein the programming rating for the particular user is associated with a graph of nodes that comprises a plurality of nodes and a plurality of edges, wherein each node in the plurality of nodes represents a respective programming subject and comprises a subject complexity rating for the particular user for the respective programming subject, said each node is connected to one or more neighbor nodes using one or more edges of the plurality of edges, wherein the one or more neighbor nodes represent one or more related programming subjects to the respective programming subject;

causing a display of the result data for the particular user on the client computer system of the particular user.

11. The server computer system of claim 10, wherein the programming rating for the particular user includes a range of values.

12. The server computer system of claim 10, wherein the complexity data comprises of a subject complexity rating for the programming task.

13. The server computer system of claim 10, wherein the programming task is associated with one or more programming subjects, and the complexity data of the programming task comprises one or more respective subject complexity ratings for the one or more programming subjects, and the one or more computer programs further comprising instructions for:

determining one or more nodes of the graph of nodes for the particular user that correspond to the one or more programming subjects; and

based on the one or more respective subject complexity ratings of the complexity data of the programming task, updating one or more corresponding subject complexity ratings of the one or more nodes of the graph of nodes for the particular user.

14. The server computer system of claim 13, the instructions further the one or more computer programs further comprising instructions for, before the updating the one or more corresponding subject complexity ratings of the one or more nodes of the graph of nodes for the particular user, based on the one or more respective subject complexity ratings of the complexity data of the programming task, adjusting the one or more respective subject complexity ratings of the complexity data according to the result data.

15. The server computer system of claim 13, the one or more computer programs further comprising instructions for, before the updating the one or more corresponding subject complexity ratings of the one or more nodes of the graph of nodes for the particular user, based on the one or more respective subject complexity ratings of the complexity data of the programming task, positively adjusting the one or more respective subject complexity ratings of the complexity data, when the result data indicates a shorter programming duration spent on the programming task by the particular user than an average programming duration for the programming task.

16. The server computer system of claim 10, wherein the programming task is associated with one or more programming subjects and the complexity data of the programming task comprises one or more respective subject complexity ratings for the one or more programming subjects, and the one or more computer programs further comprising instructions for:

determining one or more nodes of the graph of nodes for the particular user that correspond to the one or more programming subjects;

determining one or more particular neighbor nodes of the one or more nodes by traversing one or more particular edges from the one or more nodes of the graph of nodes; and

based on the one or more respective subject complexity ratings of the complexity data of the programming task, updating one or more subject complexity ratings of the one or more particular neighbor nodes of the one or more nodes of the graph of nodes for the particular user.

17. The server computer system of claim 10, wherein the programming task is associated with one or more programming subjects and the complexity data of the programming task comprises one or more respective subject complexity ratings for the one or more programming subjects, and the one or more computer programs further comprising instructions for:

determining one or more nodes of the graph of nodes for the particular user that correspond to the one or more programming subjects;

determining one or more particular neighbor nodes of the one or more nodes by traversing one or more particular edges from the one or more nodes of the graph of nodes; and

based on the one or more respective subject complexity ratings of the complexity data of the programming task, updating one or more subject complexity ratings of the one or more particular neighbor nodes of the one or more nodes of the graph of nodes for the particular user.

18. The server computer system of claim 10, the one or more computer programs further comprising instructions for determining the programming rating for the particular user by aggregating a plurality of subject complexity ratings of the plurality of nodes of the graph of nodes for the particular user using one or more aggregation functions.