Systems and methods for developing computer algorithm solutions by conducting competitions

• US 9,218,746 B2
• Filed: 03/12/2012
• Issued: 12/22/2015
• Est. Priority Date: 01/09/2001
• Status: Active Grant

• Current Assignee: TopCoder, Inc. (Wipro Limited)
• Original Assignee: Appirio Inc. (Wipro Limited)
• Inventors: Lydon, Michael, Hughes, John M.
• Primary Examiner(s): D'AGOSTINO, PAUL ANTHONY
• Application Number:
  US13/417,852
• Publication Number:
  US 20120214563A1
• Time in Patent Office: 1,380 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06Q 10/06 Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
• CPC Class Codes: G06Q 10/06398 Performance of employee with respect to a job function
• CPC Class Codes: G06Q 20/085 involving remote charge determination or related payment systems
• CPC Class Codes: G06Q 30/02 Marketing; Price estimation or determination; Fundraising
• CPC Class Codes: G06Q 30/0209 Incentive being awarded or redeemed in connection with the playing of a video game
• CPC Class Codes: G06Q 40/04 Trading; Exchange, e.g. stocks, commodities, derivatives or currency exchange
• CPC Class Codes: G07F 17/32 for games, toys, sports, or amusements
• CPC Class Codes: G07F 17/3276 wherein the players compete, e.g. tournament
• CPC Class Codes: G09B 7/00 Electrically-operated teaching apparatus or devices working with questions and answers mechanically operated G09B3/00; computing arrangements G06F

Abstract

This invention relates to a method and apparatus for providing a coding competition. In one embodiment, a method for providing a coding competition includes transmitting a coding problem to contestants, receiving computer code created by a contestant, determining a response of the computer code to test data, and evaluating the response of the computer code to the test data. In another embodiment, a method for evaluating a skill level of a contestant includes electronically communicating a coding problem to contestants, electronically receiving a software program in response to the coding problem from one of the contestants, evaluating the received software program, awarding points to the contestant based on the received software program, and determining a rating for the contestant for the competition based on the number of points awarded to the contestant.

Claims

1. A system for automatically identifying solutions to a problem statement by conducting a competition, the system comprising:

a communication module for electronically inviting via a communications network a plurality of potential participants to participate in a competition provided over the communications network, the competitions having a predetermined start time and end time, and for electronically receiving indicia of interest from potential participants;

a contest server comprising at least one memory storing computer-executable instructions, and at least one processing unit for executing the instructions stored in the memory, wherein execution of the instructions results in an application comprising;

(i) a submission receiving subsystem for electronically receiving software-based submissions from participants after the start time and prior to the end time of the competition,

(ii) a testing engine for;

(a) after the start time and prior to the end time, automatically performing interim evaluations of the received software-based submissions by executing, using a processor, each received software-based submission using first test data as input and assigning points to the participants based on the interim evaluations,

(b) for each received software-based submission from a participant, selecting respective second test data for the participant based on a performance of the received software-based submission in at least one of the interim evaluations of the received software-based submission, and

(c) after the end time, automatically performing final evaluations of the last software-based submission received from each participant by executing, using a processor, each last software-based submission using the second test data selected for the participant as input and assigning points to the participants based on the final evaluations, and

(iii) a winner identification subsystem for automatically identifying as the winner the participant who submitted the submission that received the greatest number of points during the final evaluations; and

a data store for storing the received submissions.

2. The system of claim 1, wherein the testing engine performs evaluations by providing test data as input to a reference program and the received submissions, and comparing output responses from the reference program to output responses from the received submissions.

3. The system of claim 1, wherein performing the final evaluations comprises automatically determining a response of each last submission to the second test data.

4. The system of claim 3, wherein performing the final evaluations further comprises automatically determining the correctness of the determined response using the second test data.

5. The system of claim 3, wherein performing the final evaluations further comprises automatically determining the accuracy of the determined response using the second test data.

6. The system of claim 3, wherein performing the final evaluations further comprises automatically determining the speed of the determined response by measuring the time taken to determine the response of the last submission to the second test data.

7. The system of claim 3, wherein performing the final evaluations further comprises automatically determining the accuracy of the determined response and the speed of the determined response using the second test data.

8. The system of claim 3, wherein performing the final evaluations further comprises automatically determining the the correctness of the determined response, the accuracy of the determined response, and the speed of the determined response using the second test data.

9. The system of claim 1, wherein the second test data comprises the first test data and additional test data.

10. The system of claim 1, wherein at least one of the first test data and the second test data comprises challenge test data submitted by at least one of another participant, a judge, and a spectator.

11. A method, implemented on at least one computer, for automatically identifying solutions to a problem statement by conducting a competition, the at least one computer comprising:

at least one memory storing computer-executable instructions; and

at least one processing unit for executing the instructions stored in the memory, wherein the execution of the instructions results in the at least one computer performing the steps of;

electronically inviting via a communications network a plurality of potential participants to participate in a competition provided over the communications network, the competitions having a predetermined start time and end time, and for electronically receiving indicia of interest from potential participants,

electronically receiving software-based submissions from participants after the start time and prior to the end time of the competition,

after the start time and prior to the end time, automatically performing interim evaluations of the received software-based submissions by executing each received software-based submission using first test data as input and assigning points to the participants based on the interim evaluations,

for each received software-based submission from a participant, selecting respective second test data for the participant based on a performance of the received software-based submission in at least one of the interim evaluations of the received software-based submission,

after the end time, automatically performing final evaluations of the last software-based submission received from each participant by executing each last software-based submission using the second test data selected for the participant as input and assigning points to the participants based on the final evaluations,

automatically identifying as the winner the participant who submitted the submission that received the greatest number of points during the final evaluations, and

storing the received submissions.

12. The method of claim 11, wherein evaluations are performed by providing test data as input to a reference program and the received submissions, and comparing output responses from the reference program to output responses from the received submissions.

13. The method of claim 11, wherein performing the final evaluations comprises automatically determining a response of each last submission to the second test data.

14. The method of claim 13, wherein performing the final evaluations further comprises automatically determining the correctness of the determined response using the second test data.

15. The method of claim 13, wherein performing the final evaluations further comprises automatically determining the accuracy of the determined response using the second test data.

16. The method of claim 13, wherein performing the final evaluations further comprises automatically determining the speed of the determined response by measuring the time taken to determine the response of the last submission to the second test data.

17. The method of claim 13, wherein performing the final evaluations further comprises automatically determining the accuracy of the determined response and the speed of the determined response using the second test data.

18. The method of claim 13, wherein performing the final evaluations further comprises automatically determining the the correctness of the determined response, the accuracy of the determined response, and the speed of the determined response using the second test data.

19. The method of claim 11, wherein the second test data comprises the first test data and additional test data.

20. The method of claim 11, wherein at least one of the first test data and the second test data comprises challenge test data submitted by at least one of another participant, a judge, and a spectator.