Automated validation of results of human performance of tasks

• US 8,498,892 B1
• Filed: 03/26/2012
• Issued: 07/30/2013
• Est. Priority Date: 11/16/2004
• Status: Expired due to Fees

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Cohen, Peter D., Nichols, Isaac E., Bravenec, John B.
• Primary Examiner(s): Jeanty, Romain
• Application Number:
  US13/430,539
• Time in Patent Office: 491 Days
• Field of Search: 705/7.38
• US Class Current: 705/7.38
• CPC Class Codes: G06Q 10/06 Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
• CPC Class Codes: G06Q 10/06311 Scheduling, planning or task assignment for a person or group
• CPC Class Codes: G06Q 30/08 Auctions

Abstract

Techniques are described for facilitating automated validation of results of performance of tasks. In some situations, task requesters may specify various automated validation criteria for tasks that they submit, and those specified validation criteria may automatically be used to determine whether results of performance of those tasks by task performers are validated. To facilitate such automated validation, predefined types of validation criteria may be provided that a user may select and/or configure. The outcome of automated validation of results may be used in various ways, such as to indicate when results have been validated and/or have not been validated, to determine whether to provide results to a task requester (e.g., if results are not provided unless they have been validated, or if results that fail validation are corrected or replaced with results that pass validation) and/or to compensate task performers for providing the results.

Claims

1. A computer-implemented method comprising:

receiving, by one or more configured computing systems, information from a task requester about a task that is available for performance and that has one or more associated validation tests;

receiving, by the one or more configured computing systems, one or more results from performance of the task by one or more task performers;

applying, by the one or more configured computing systems, at least one of the validation tests associated with the task to the received one or more results;

determining, by the one or more configured computing systems, that the received one or more results are validated based at least in part on the received one or more results passing the applied validation tests; and

providing the received one or more results to the task requester.

2. The method of claim 1 wherein the at least one validation test includes obtaining multiple distinct results from performance of the task by multiple task performers, and wherein the applying of the at least one validation test includes determining that a result is validated if the result is in agreement with a majority of the multiple distinct results.

3. The method of claim 1 wherein the at least one validation test includes obtaining multiple distinct results from performance of the task by multiple task performers, and wherein the applying of the at least one validation test includes determining that a result is validated if the result is in agreement with a specified number of the multiple distinct results.

4. The method of claim 1 wherein the at least one validation test includes comparing one or more results from performance of the task to other results from performance of other related tasks in order to determine whether the results being compared are similar.

5. The method of claim 1 wherein the at least one validation test includes determining if one or more results from performance of the task are each of a specified type of data.

6. The method of claim 1 wherein the at least one validation test includes determining if one or more results from performance of the task are each in accordance with a specified format.

7. The method of claim 1 wherein the at least one validation test includes determining if one or more results from performance of the task match one or more specified allowable values for the results.

8. The method of claim 1 wherein the at least one validation test includes executing code supplied by the task requester who submitted the task.

9. The method of claim 1 further comprising receiving results from performance of one or more other tasks that do not have associated validation tests, and providing the received results to a task requester who submitted the other tasks without validating the received results.

10. The method of claim 1 further comprising:

obtaining results from performance of another task that is submitted by an indicated task requester and that has one or more other associated validation tests;

determining that the obtained results are not validated based at least in part on the obtained results not passing the one or more other associated validation tests; and

providing information to the indicated task requester that includes the obtained results and an indication of a lack of validation of the obtained results.

11. The method of claim 1 wherein the providing of the received one or more results includes providing an indication of validation of those results.

12. The method of claim 1 further comprising providing payment to the one or more task performers who performed the task based on the received one or more results being determined to be validated.

13. The method of claim 1 wherein the one or more configured computing systems are part of a task fulfillment system, and wherein the task requester and the one or more task performers are users of the task fulfillment system.

14. A non-transitory computer-readable medium whose contents configure a computing device to perform a method comprising:

receiving an indication from a user of one or more tasks that are not yet performed;

providing information to the user to enable the user to indicate at least one predefined type of validation criteria that is available for validating results of task performance; and

after receiving an indication from the user to use one or more of the at least one predefined types of validation criteria in an indicated manner, using the indicated one or more predefined types of validation criteria in the indicated manner to validate results of performance of the one or more tasks.

15. The non-transitory computer-readable medium of claim 14 wherein the user is a task requester who is submitting the one or more tasks as each being available for performance by one or more task performers via a task fulfillment system provided by the configured computing device, and wherein the providing of the information to the user includes selecting the at least one predefined types of validation criteria to be indicated to the user based at least in part on the one or more tasks.

16. The non-transitory computer-readable medium of claim 14 wherein the receiving of the indication from the user to use the one or more predefined types of validation criteria in an indicated manner includes defining one or more validation criteria based on information provided by the user.

17. The non-transitory computer-readable medium of claim 16 wherein the providing of the information to the user includes indicating multiple predefined types of validation criteria, at least one of the predefined types of validation criteria being configurable, and wherein the defining of one of the defined validation criteria includes using information supplied by the user for configuration of one of the configurable predefined types.

18. The non-transitory computer-readable medium of claim 14 wherein the contents include one or more data structures for use in validating the results of performance of the one or more tasks, the data structure comprising one or more entries that correspond to the one or more tasks and contain information that includes the indicated one or more predefined types of validation criteria for use in validating the results of performance of the one or more tasks.

20. The non-transitory computer-readable medium of claim 14 wherein the computer-readable medium is a memory of the configured computing device, wherein the contents are instructions that when executed program the computing device to perform the method, and wherein the method further comprises:

after the receiving from the user of the indication of the one or more predefined types of validation criteria, receiving the results of the performance of the one or more tasks, each of the one or more tasks being performed by one or more task performers; and

after the results of the performance of each of the one or more tasks are validated based on the indicated one or more predefined types of validation criteria, providing payment to the one or more task performers who performed the task based at least in part on the results being validated.

19. A non-transitory computer-readable medium whose contents configure a computing device to perform a method comprising:

receiving an indication from a user of one or more tasks that are not yet performed;

providing information to the user to enable the user to indicate at least one predefined type of validation criteria that is available for validating results of task performance; and

after receiving an indication from the user to use one or more of the at least one predefined types of validation criteria in an indicated manner, using, by the configured computing device, the indicated one or more predefined types of validation criteria in the indicated manner to validate results of performance of the one or more tasks, wherein the using of the indicated one or more predefined types of validation criteria includes, after receiving one or more results from performance of the one or more tasks by one or more task performers, automatically determining whether the received one or more results are validated based at least in part on whether the received one or more results satisfy the one or more predefined types of validation criteria.

21. A computing device configured to validate results of task performance, comprising:

a processor;

a first component that is configured to, when executed by the processor, receive information about a task that is available from a task requester and that has one or more associated validation criteria; and

a second component that is configured to, when executed by the processor and after receiving one or more results from performance of the task by one or more task performers;

determine to initiate applying the one or more associated validation criteria for the task to the received one or more results; and

determine whether the received one or more results are validated based at least in part on the applying of the one or more associated validation criteria for the task.

22. The computing device of claim 21 wherein the second component is further configured to, if it is determined that the received one or more results for the task are validated, provide the received results and an indication of the validation of the received results to a task requester who supplied the task.

23. The computing device of claim 21 wherein the one or more associated validation criteria include multiple validation tests, and wherein the determining of to initiate applying the one or more associated validation criteria for the task includes automatically performing multiple of the validation tests.

24. The computing device of claim 21 further comprising a third component that is configured to, after the received one or more results are determined to be validated, providing payment to the one or more task performers based at least in part on the received results being validated.

25. The computing device of claim 21 wherein the first and second components each include software instructions for execution by the processor of the computing device.