Automated evaluation of computer programming

• US 10,303,517 B1
• Filed: 01/27/2017
• Issued: 05/28/2019
• Est. Priority Date: 01/28/2016
• Status: Active Grant

• Current Assignee: Codesignal Incorporated
• Original Assignee: BrainFights, Inc.
• Inventors: Sloyan, Tigran, Shatakhtsyan, Aram, Desroches, Jan-Felix, Piliposyan, Eduard
• Primary Examiner(s): Kessler, Gregory A
• Application Number:
  US15/418,371
• Time in Patent Office: 851 Days
• Field of Search: None
• CPC Class Codes: G06F 11/3616 using software metrics
• CPC Class Codes: G06F 11/3672 Test management
• CPC Class Codes: G06F 11/3688 for test execution, e.g. scheduling of test suites
• CPC Class Codes: G06F 11/3692 for test results analysis
• CPC Class Codes: G06F 8/30 Creation or generation of source code
• CPC Class Codes: G06F 8/71 Version control security arrangements therefor G06F21/57; Configuration management
• CPC Class Codes: G06F 8/77 Software metrics
• CPC Class Codes: G06F 9/505 considering the load

Abstract

Techniques are described for analysis and evaluation of different computer programs developed by different users to accomplish programming tasks. A computer programming evaluation system provides users with one or more programming tasks and ability to input source code to solve the programming tasks. In an embodiment, using a client computer system, a user initiates a computer programming session with a programming system. As a result of the session, the client computer system is provided with user interface elements allowing the user to view one or more programming tasks and submit to the programming system source code for the one or more programming tasks. In an embodiment, different modes are used by the programming system to evaluate quality of the computer programming performed by a user in terms of accomplishing the programming task. By collecting result metrics from evaluations and comparing with complexity data of the programming tasks, the programming system generates a graph-based quantified knowledge map of computer programming for a user. Non-limiting examples of modes for evaluation of the computer programming include a practice mode, an arcade mode, versus mode, a bot mode and a bot training mode. In an embodiment, to execute source codes in parallel, a programming system may include multiple programming servers. Each programming server is a separate computer system that, in some embodiments, is a cloud computer system allocated from cloud services. Executing different instances of source code on different programming servers improves both the security and scalability of the programming system.

Claims

1. A computer-implemented method comprising:

establishing multiple con-current programming sessions initiated by a plurality of client computer systems;

receiving a first source code execution request from a first con-current programming session of the multiple con-current programming sessions, the first source code execution request comprising a first instances of source code;

receiving a second source code execution request from a second con-current programming session of the multiple con-current programming sessions, the second source code execution request comprising a second instance of source code;

determining, based at least on load balancing of source code execution requests among a plurality of programming computer servers, to process on a first programming computer server, of the plurality of programming computer servers, the first source code execution request from the first con-current programming session and to process on a second programming computer server, of the plurality of programming computer servers, the second source code execution request from the second con-current programming session;

executing the first instance of source code by the first programming computer server concurrently with executing the second instance of source code by the second programming computer server, wherein, regardless of content of the first instance of source code, the executing the first instance of source code by the first programming computer server can only access a configured set of resources;

monitoring one or more server selectability metrics for the plurality of programming computer servers;

based on the one or more server selectability metrics for the plurality of programming computer servers, determining to deploy a new programming computer server;

deploying the new programming computer server based on an image of a specific programming computer server;

wherein the specific programming computer server has been previously determined to be stable.

2. The method of claim 1, wherein the determining to execute the first source code execution request on the first programming computer server causes selecting the second programming computer server, of the plurality of programming computer servers, to process the second source code execution request from the second con-current programming session.

3. The method of claim 1, further comprising:

monitoring one or more server selectability metrics for the first programming computer server;

comparing the one or more server selectability metrics of the first programming computer server to one or more server selectability metrics of one or more other programming computer servers of the plurality of programming computer servers;

based on the comparing, determining to process the first source code execution request on the first programming computer server.

4. The method of claim 3, wherein the one or more server selectability metrics for the first programming computer server are based on one or more of:

a current number of source code requests in a server queue of source code requests for the first programming computer server, an average number of source code requests in the server queue of source code requests for the first programming computer server, a length of time period for returning a result for one or more latest source code execution requests or an error rate of execution of instances of source code on the first programming computer server.

5. The method of claim 1, wherein each of the plurality of programming computer servers is configured to execute specific one or more types of source code.

6. The method of claim 1, further comprising:

maintaining a first queue of the source code execution requests to be processed by the first programming computer server, wherein each entry in the first queue is processed in a specified order of insertion;

in response to determining, based at least on load balancing of the source code execution requests among the plurality of programming computer servers, to process on the first programming computer server, of the plurality of programming computer servers, the first source code execution request, inserting a particular entry for the first source code execution request into the first queue;

wherein prior to the inserting the particular entry into the first queue, the first queue already contains one or more previous entries for one or more previously received source code execution requests; and

performing the executing the first instance of source code by the first programming computer server after the one or more previously received source code execution requests have been processed by the first programming computer server.

7. The method of claim 1 further comprising:

maintaining a plurality of template images of run-time environments, wherein each template image is configured to execute one or more types of source code;

based on a particular type of the first instance of source code, determining to select a particular template image of a particular run-time environment from the plurality of template images;

executing the particular template image to setup the particular run-time environment on the first programming computer server.

8. The method of claim 7, wherein the particular template image stores instructions to configure the particular run-time environment to have only access to the configured set of resources.

9. The method of claim 7, wherein the particular template image stores instructions to configure the particular run-time environment with one or more compilation tools for compiling the particular type of the first instance of source code.

10. The method of claim 7, further comprising:

executing the first instance of source code by the first programming computer server by executing the first instance of source code in the particular run-time environment;

in response to a request to stop executing the first instance of source code, terminating execution of the particular run-time environment which causes the executing the first instance of source code by the first programming computer server to stop.

11. The method of claim 7, further comprising:

executing the first instance of source code by the first programming computer server by executing the first instance of source code in the particular run-time environment;

in response to completion of the executing the first instance of source code in the particular run-time environment, terminating execution of the particular run-time environment which causes the executing the first instance of source code by the first programming computer server to stop.

12. The method of claim 7, wherein:

the particular type of the first instance of source code is a data base query,

the particular template image includes instructions to configure a database management system,

the first source code execution request includes one or more queries to set up one or more database objects;

the method further comprises;

executing the particular template image to setup the particular run-time environment on the first programming computer server causing the database management system to run in the particular run-time environment;

executing the first instance of source code by the first programming computer server at least by requesting the database management system to execute the one or more queries to setup the one or more database objects.

13. A computer-implemented method comprising:

establishing multiple con-current programming sessions initiated by a plurality of client computer systems;

receiving a first source code execution request from a first con-current programming session of the multiple con-current programming sessions, the first source code execution request comprising a first instances of source code;

receiving a second source code execution request from a second con-current programming session of the multiple con-current programming sessions, the second source code execution request comprising a second instance of source code;

determining, based at least on load balancing of source code execution requests among a plurality of programming computer servers, to process on a first programming computer server, of the plurality of programming computer servers, the first source code execution request from the first con-current programming session and to process on a second programming computer server, of the plurality of programming computer servers, the second source code execution request from the second con-current programming session;

executing the first instance of source code by the first programming computer server concurrently with executing the second instance of source code by the second programming computer server, wherein, regardless of content of the first instance of source code, the executing the first instance of source code by the first programming computer server can only access a configured set of resources;

maintaining a queue of the source code execution requests to be processed by the plurality of programming computer servers, wherein each entry in the queue is processed in an order of insertion into the queue;

after receiving the first source code execution request from the first con-current programming session of the multiple con-current programming sessions, inserting an entry into the queue for the first source code execution request;

after receiving the second source code execution request from the second con-current programming session of the multiple con-current programming sessions, inserting an entry into the queue for the second source code execution request.

14. The method of claim 13, wherein the determining to execute the first source code execution request on the first programming computer server causes selecting the second programming computer server, of the plurality of programming computer servers, to process the second source code execution request from the second con-current programming session.

15. The method of claim 13, further comprising:

monitoring one or more server selectability metrics for the first programming computer server;

comparing the one or more server selectability metrics of the first programming computer server to one or more server selectability metrics of one or more other programming computer servers of the plurality of programming computer servers;

based on the comparing, determining to process the first source code execution request on the first programming computer server.

16. The method of claim 15, wherein the one or more server selectability metrics for the first programming computer server are based on one or more of:

a current number of source code requests in a server queue of source code requests for the first programming computer server, an average number of source code requests in the server queue of source code requests for the first programming computer server, a length of time period for returning a result for one or more latest source code execution requests or an error rate of execution of instances of source code on the first programming computer server.

17. The method of claim 13, further comprising:

maintaining a plurality of template images of run-time environments, wherein each template image is configured to execute a one or more types of source code;

based on a particular type of the first instance of source code, determining to select a particular template image of a particular run-time environment from the plurality of template images;

executing the particular template image to setup the particular run-time environment on the first programming computer server.

18. A server computer system comprising:

one or more processors;

one or more storage media storing one or more computer programs for execution by the one or more processors, the one or more computer programs comprising a set of instructions for;

establishing multiple con-current programming sessions initiated by a plurality of client computer systems;

receiving a first source code execution request from a first con-current programming session of the multiple con-current programming sessions, the first source code execution request comprising a first instances of source code;

receiving a second source code execution request from a second con-current programming session of the multiple con-current programming sessions, the second source code execution request comprising a second instance of source code;

determining, based at least on load balancing of source code execution requests among a plurality of programming computer servers, to process on a first programming computer server, of the plurality of programming computer servers, the first source code execution request from the first con-current programming session and to process on a second programming computer server, of the plurality of programming computer servers, the second source code execution request from the second con-current programming session;

executing the first instance of source code by the first programming computer server concurrently with executing the second instance of source code by the second programming computer server, wherein, regardless of content of the first instance of source code, the executing the first instance of source code by the first programming computer server can only access a configured set of resources;

monitoring one or more server selectability metrics for the plurality of programming computer servers;

based on the one or more server selectability metrics for the plurality of programming computer servers, determining to deploy a new programming computer server;

deploying the new programming computer server based on an image of a specific programming computer server;

wherein the specific programming computer server has been previously determined to be stable.

19. The server computer system of claim 18, wherein the set of instructions includes instructions for determining to execute the first source code execution request on the first programming computer server cause selecting the second programming computer server, of the plurality of programming computer servers, to process the second source code execution request from the second con-current programming session.

20. The server computer system of claim 18, the set of instructions includes instructions for:

monitoring one or more server selectability metrics for the first programming computer server;

comparing the one or more server selectability metrics of the first programming computer server to one or more server selectability metrics of one or more other programming computer servers of the plurality of programming computer servers;

based on the comparing, determining to process the first source code execution request on the first programming computer server.

21. The server computer system of claim 20, wherein the one or more server selectability metrics for the first programming computer server are based on one or more of:

a current number of source code requests in the server queue of source code requests for the first programming computer server, an average number of source code requests in the server queue of source code requests for the first programming computer server, a length of time period for returning a result for one or more latest source code execution requests or error rate of execution of instances of source code on the first programming computer server.

22. A server computer system comprising:

one or more processors;

one or more storage media storing one or more computer programs for execution by the one or more processors, the one or more computer programs comprising a set of instructions for;

establishing multiple con-current programming sessions initiated by a plurality of client computer systems;

receiving a first source code execution request from a first con-current programming session of the multiple con-current programming sessions, the first source code execution request comprising a first instances of source code;

receiving a second source code execution request from a second con-current programming session of the multiple con-current programming sessions, the second source code execution request comprising a second instance of source code;

determining, based at least on load balancing of source code execution requests among a plurality of programming computer servers, to process on a first programming computer server, of the plurality of programming computer servers, the first source code execution request from the first con-current programming session and to process on a second programming computer server, of the plurality of programming computer servers, the second source code execution request from the second con-current programming session;

executing the first instance of source code by the first programming computer server concurrently with executing the second instance of source code by the second programming computer server, wherein, regardless of content of the first instance of source code, the executing the first instance of source code by the first programming computer server can only access a configured set of resources;

maintaining a queue of the source code execution requests to be processed by the plurality of programming computer servers, wherein each entry in the queue is processed in an order of insertion into the queue;

after receiving the first source code execution request from the first con-current programming session of the multiple con-current programming sessions, inserting an entry into the queue for the first source code execution request;

after receiving the second source code execution request from the second con-current programming session of the multiple con-current programming sessions, inserting an entry into the queue for the second source code execution request.

23. The server computer system of claim 22, wherein the set of instructions includes instructions for determining to execute the first source code execution request on the first programming computer server causes selecting the second programming computer server, of the plurality of programming computer servers, to process the second source code execution request from the second con-current programming session.

24. The server computer system of claim 22, wherein the set of instructions includes instructions for:

monitoring one or more server selectability metrics for the first programming computer server;

comparing the one or more server selectability metrics of the first programming computer server to one or more server selectability metrics of one or more other programming computer servers of the plurality of programming computer servers;

based on the comparing, determining to process the first source code execution request on the first programming computer server.

25. The server computer system of claim 24, wherein the one or more server selectability metrics for the first programming computer server are based on one or more of:

a current number of source code requests in a server queue of source code requests for the first programming computer server, an average number of source code requests in the server queue of source code requests for the first programming computer server, a length of time period for returning a result for one or more latest source code execution requests or an error rate of execution of instances of source code on the first programming computer server.

26. The server computer system of claim 22, wherein the set of instructions includes instructions for:

maintaining a plurality of template images of run-time environments, wherein each template image is configured to execute one or more types of source code;

based on a particular type of the first instance of source code, determining to select a particular template image of a particular run-time environment from the plurality of template images;

executing the particular template image to setup the particular run-time environment on the first programming computer server.