Automated validation of results of human performance of tasks

• US 8,170,897 B1
• Filed: 09/26/2006
• Issued: 05/01/2012
• Est. Priority Date: 11/16/2004
• Status: Expired due to Fees

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Cohen, Peter D., Nichols, Isaac E., Bravenec, John B.
• Primary Examiner(s): Jeanty, Romain
• Application Number:
  US11/535,388
• Time in Patent Office: 2,044 Days
• Field of Search: 705/11, 705/7.26
• US Class Current: 705/7.13
• CPC Class Codes: G06Q 10/06 Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
• CPC Class Codes: G06Q 10/06311 Scheduling, planning or task assignment for a person or group
• CPC Class Codes: G06Q 30/08 Auctions

Abstract

Techniques are described for facilitating automated validation of results of performance of tasks. In some situations, task requesters may specify various automated validation criteria for tasks that they submit, and those specified validation criteria may automatically be used to determine whether results of performance of those tasks by task performers are validated. To facilitate such automated validation, predefined types of validation criteria may be provided that a user may select and/or configure. The outcome of automated validation of results may be used in various ways, such as to indicate when results have been validated and/or have not been validated, to determine whether to provide results to a task requester (e.g., if results are not provided unless they have been validated, or if results that fail validation are corrected or replaced with results that pass validation) and/or to compensate task performers for providing the results.

Claims

1. A computer-implemented method for providing an electronic marketplace that acts as an intermediary to facilitate performance by task performer users of tasks supplied by task requesters, the method comprising:

for each of multiple tasks, receiving information about the task from a task requester user who is submitting the task as being available for performance by one or more task performer users, at least some of the multiple tasks each having one or more associated validation tests that are specified by the task requester user who submitted the task, the one or more associated validation tests for one of the at least some tasks including obtaining initial results from one or more initial task performer users and receiving an evaluation of the initial results from one or more other task performer users that are more highly qualified than the initial task performer users, such that success of those one or more associated validation tests includes that the evaluation is positive;

automatically facilitating performance of at least some of the multiple submitted tasks by providing information about tasks to task performer users and receiving results of performance of tasks by task performer users, the received results including results for one or more of the at least some tasks, the automatic facilitating being performed by one or more configured computing systems that are part of the electronic marketplace; and

for each of the one or more of the at least some tasks, after the receiving of the results of performance of the task by one or more task performer users,

automatically performing the one or more associated validation tests on the results for the task and determining that the results are acceptable based at least in part on the performed validation tests succeeding, the automatic performing of the one or more associated validation tests being performed by the one or more configured computing systems; and

in response to determining that the results are acceptable, providing the results to the task requester user who submitted the task.

2. The method of claim 1 wherein, for one of the at least some tasks, the receiving of the information about the one task from a task requester user includes receiving one or more specifications from the task requester user of the one or more associated validation tests for the one task.

3. The method of claim 2 wherein the receiving of the one or more specifications from the task requester user of the one or more associated validation tests for the one task includes providing information to the task requester user indicating multiple predefined types of validation tests, and receiving information from the task requester user indicating, for each of the one or more associated validation tests for the one task, one of the predefined validation test types and one or more configurations for the indicated one validation test type.

4. The method of claim 2 further comprising, before the receiving of the information about the one task from the task requester user, defining one or more validation tests for an indicated task type based on information received from the task requester user, and wherein the information received from the task requester user of the one or more associated validation tests for the one task is an indication that the one task is of the indicated task type.

5. The method of claim 1 further comprising, determining that the results of performance of one of the at least some tasks are unacceptable, performing further processing to obtain acceptable results for the one task, and providing the obtained acceptable results for the one task to the task requester user who submitted the one task.

6. The method of claim 5 wherein the further processing to obtain the acceptable results for the one task includes obtaining corrections to the unacceptable results from one or more task performer users other than the task performer users who provided the unacceptable results.

7. The method of claim 5 wherein the determining that the results of performance of the one task are unacceptable is based on any of the validation tests for the one task not being satisfied.

8. The method of claim 1 wherein, for one of the at least some tasks, the one or more associated validation tests for the one task include obtaining multiple results from multiple task performer users who independently perform the one task, such that success of those one or more associated validation tests includes obtaining an indicated degree of agreement among the multiple results.

9. The method of claim 1 further comprising receiving results of performance of a task that does not have any associated validation tests, and providing the received results to the task requester user who submitted the task without performing automated validation of the received results.

10. A computer-implemented method comprising:

receiving information from a task requester about one or more submitted tasks that are each available for performance by one or more task performers and that each have one or more associated validation tests, the one or more validation tests associated with one of the submitted tasks being specified by the task requester who submitted the one task, the specifying being via interactions with a user interface provided by one or more configured computing systems acting as an intermediary between task requester users and task performer users to facilitate performance of tasks; and

for each of at least one of the submitted tasks,

receiving one or more results from performance of the task by one or more task performers;

automatically applying at least one of the validation tests associated with the task to the received one or more results, the automatic applying being performed by the one or more configured computing systems;

automatically determining that the received one or more results are validated based at least in part on the received one or more results passing the applied validation tests, the automatic determining being performed by the one or more configured computing systems; and

providing the received one or more results to the task requester.

11. The method of claim 10 wherein one of the validation tests for one of the submitted tasks has one or more associated actions to be performed, and wherein the applying of the one validation test includes automatically performing the one or more associated actions.

12. The method of claim 10 wherein one of the validation tests for one of the submitted tasks includes obtaining multiple distinct results from performance of the one task by multiple task performers such that a result is validated if the result is in agreement with a majority of the multiple distinct results.

13. The method of claim 10 wherein one of the validation tests for one of the submitted tasks includes obtaining multiple distinct results from performance of the one task by multiple task performers such that a result is validated if the result is in agreement with an indicated plurality of the multiple distinct results.

14. The method of claim 10 wherein one of the validation tests for one of the submitted tasks includes obtaining multiple distinct results from performance of the one task by multiple task performers such that a result is validated if the result is in agreement with a specified number of the multiple distinct results.

15. The method of claim 10 wherein one of the validation tests for one of the submitted tasks includes automatically comparing one or more results from performance of the one task to other results from performance of other related tasks in order to determine whether the results being compared are sufficiently similar.

16. The method of claim 10 wherein one of the validation tests for one of the submitted tasks includes automatically determining if one or more results from performance of the one task are each of a specified type of data.

17. The method of claim 10 wherein one of the validation tests for one of the submitted tasks includes automatically determining if one or more results from performance of the one task are each in accordance with a specified format.

18. The method of claim 10 wherein one of the validation tests for one of the submitted tasks includes automatically determining if one or more results from performance of the one task each match one or more specified allowable values.

19. The method of claim 18 wherein the one or more allowable values are specified using at least one of a mathematical expression, a relational expression, a logical expression, and a programming expression.

20. The method of claim 10 wherein one of the validation tests for one of the submitted tasks includes automatically executing code supplied by the task requester who submitted the one task.

21. The method of claim 10 wherein the method is performed on behalf of an electronic marketplace that facilitates performance of tasks, and wherein the one or more associated validation tests for one of the submitted tasks are indicated programmatically via an application programming interface provided by the electronic marketplace.

22. The method of claim 10 further comprising receiving results from performance of one or more other tasks that do not have associated validation tests, and providing the received results to a task requester who submitted the other tasks without automatically validating the received results.

23. The method of claim 10 further comprising, for one of the submitted tasks that is distinct from the at least one tasks, receiving one or more results from performance of the one task by one or more task performers, automatically determining that received one or more results for the one task are not validated, and providing an indication to the task requester of the received one or more results for the one task and of a lack of validation of those results.

24. The method of claim 10 wherein the providing of the received one or more results for each of the at least one submitted tasks includes providing an indication of validation of those results.

25. The method of claim 10 wherein the determining that the received one or more results for one of the submitted tasks are validated based on those results passing the applied validation tests for the one task includes determining that those results are validated based on those results passing all of the validation tests associated with the one task.

26. The method of claim 10 further comprising, for each of one or more of the at least one submitted tasks, automatically providing payment to the one or more task performers who performed the task.

27. A computer-implemented method comprising:

receiving information from a task requester about one or more submitted tasks that are each available for performance by one or more task performers and that each have one or more associated validation tests, wherein one of the submitted tasks further has one or more associated validation criteria other than the one or more validation tests associated with that one task; and

for each of at least one of the submitted tasks,

receiving one or more results from performance of the task by one or more task performers;

automatically applying at least one of the validation tests associated with the task to the received one or more results, the automatic applying being performed by one or more configured computing systems;

automatically determining that the received one or more results are validated based at least in part on the received one or more results passing the applied validation tests, the automatic determining being performed by the one or more configured computing systems, and wherein the determining is further based at least in part on the received one or more results for the one task satisfying the one or more associated validation criteria; and

providing the received one or more results to the task requester.

28. A computer-implemented method comprising:

receiving information from a task requester about one or more submitted tasks that are each available for performance by one or more task performers and that each have one or more associated validation tests; and

for each of at least one of the submitted tasks,

receiving one or more results from performance of the task by one or more task performers;

automatically applying at least one of the validation tests associated with the task to the received one or more results, the automatic applying being performed by one or more configured computing systems;

automatically determining that the received one or more results are validated based at least in part on the received one or more results passing the applied validation tests, the automatic determining being performed by the one or more configured computing systems; and

providing the received one or more results to the task requester,

and wherein one of the validation tests for one of the submitted tasks includes obtaining a verification of one or more results received from performance of the one task by one or more task performers, the verification being from one or more other task performers that did not provide the one or more received results.

29. The method of claim 28 wherein the one or more task performers who provide the one or more received results for the one task have associated qualifications specified for the one task, and wherein the one or more other task performers have other qualifications specified by the one validation test.

30. The method of claim 28 wherein the verification of the one or more results includes an evaluation that the results satisfy one or more specified criteria, and wherein the one or more other task performers who provide the evaluation are more highly qualified than the one or more task performers who provide the results.

31. A computer-implemented method comprising:

receiving information from a task requester about one or more submitted tasks that are each available for performance by one or more task performers and that each have one or more associated validation tests, wherein one of the submitted tasks has multiple associated validation tests and further includes one or more instructions for use of the multiple associated validation tests; and

for each of at least one of the submitted tasks,

receiving one or more results from performance of the task by one or more task performers;

automatically applying at least one of the validation tests associated with the task to the received one or more results, the automatic applying being performed by one or more configured computing systems, and wherein the applying of the at least one validation tests associated with the one task having the multiple associated validation tests includes applying one or more of the multiple associated validation tests in accordance with the included instructions;

automatically determining that the received one or more results are validated based at least in part on the received one or more results passing the applied validation tests, the automatic determining being performed by the one or more configured computing systems; and

providing the received one or more results to the task requester.

32. The method of claim 31 wherein the one or more instructions provide logic for use in determining how to apply the multiple associated validation tests based on conditions at a time of the applying.

33. The method of claim 31 wherein the one or more instructions provide logic for use in determining a sequence that the multiple associated validation tests are to be applied.

34. The method of claim 31 wherein the one or more instructions provide logic for use in selecting at least one of the multiple validation tests to use based at least in part on attributes of one or more task performers who provide one or more results for the one task.

35. A computer-implemented method comprising:

presenting a user interface to users that provides information about multiple predefined types of validation tests and that provides capabilities for the users to interactively select and configure one or more of the validation test types for one or more tasks;

receiving information from a task requester about one or more submitted tasks that are each available for performance by one or more task performers and that each have one or more associated validation tests, and wherein at least one of the validation tests for at least one of the submitted tasks is specified based on interactions of a task requester user with the presented user interface; and

for each of at least one of the submitted tasks,

receiving one or more results from performance of the task by one or more task performers;

automatically applying at least one of the validation tests associated with the task to the received one or more results, the automatic applying being performed by one or more configured computing systems;

automatically determining that the received one or more results are validated based at least in part on the received one or more results passing the applied validation tests, the automatic determining being performed by the one or more configured computing systems; and

providing the received one or more results to the task requester.

36. A non-transitory computer-readable medium whose contents configure a computing device to automatically validate results of task performance, by performing a method comprising:

receiving an indication from a user of one or more tasks that are not yet performed, the receiving of the indication of the one or more tasks including receiving an indication of a type of task such that the one or more tasks are of the indicated task type;

providing information to the user to indicate at least one predefined type of validation criteria that is available; and

after receiving an indication from the user to use one or more of the at least one predefined types of validation criteria in an indicated manner, using the indicated one or more predefined types of validation criteria in the indicated manner to automatically validate results of performance of the one or more tasks, the using of the indicated one or more predefined types of validation criteria including storing the indicated one or more predefined types of validation criteria for tasks of the indicated task type that are later submitted by the user.

37. The non-transitory computer-readable medium of claim 36 wherein the method further comprises:

after the receiving from the user of the indication of the one or more predefined types of validation criteria, receiving the results of the performance of the one or more tasks, each of the one or more tasks being performed by one or more task performers; and

after the results of the performance of each of the one or more tasks are automatically validated based on the indicated one or more predefined types of validation criteria, automatically providing payment to the one or more task performers who performed the task based at least in part on the results being validated.

38. The non-transitory computer-readable medium of claim 36 wherein the computer-readable medium is a memory of the configured computing device.

39. The non-transitory computer-readable medium of claim 36 wherein the contents are instructions that when executed program the configured computing device to perform the method.

40. The non-transitory computer-readable medium of claim 36 wherein the contents include one or more data structures for validating results of performance of a task, the data structure comprising multiple entries that each correspond to a task and contain information that includes one or more validation criteria.

41. A non-transitory computer-readable medium whose contents configure a computing device to automatically validate results of task performance, by performing a method comprising:

receiving an indication from a user of one or more tasks that are not yet performed;

providing information to the user to indicate at least one predefined type of validation criteria that is available; and

after receiving an indication from the user to use one or more of the at least one predefined types of validation criteria in an indicated manner, using the indicated one or more predefined types of validation criteria in the indicated manner to automatically validate results of performance of the one or more tasks,

and wherein the receiving of the indication from the user to use the one or more predefined types of validation criteria in the indicated manner includes defining one or more validation criteria that are each of one or more of the indicated at least one predefined types, and wherein the using of the indicated one or more predefined types of validation criteria includes, after receiving one or more results from performance of the task by one or more task performers, automatically determining whether the received one or more results are validated based at least in part on whether the received one or more results satisfy the one or more defined validation criteria.

42. The non-transitory computer-readable medium of claim 41 wherein the user is a task requester who is submitting the one or more tasks as each being available for performance by one or more task performers, and wherein the providing of the information to the user is performed after the receiving of the indication from the user of the one or more tasks that are not yet performed, and includes selecting the at least one predefined types of validation criteria to be indicated to the user based at least in part on the one or more tasks.

43. The non-transitory computer-readable medium of claim 41 wherein the providing of the information to the user includes indicating multiple predefined types of validation criteria, at least one of the predefined types of validation criteria being configurable so as to enable multiple validation criteria of that predefined type to be created, and wherein the defining of one of the defined validation criteria includes using information supplied by the user for configuration of one of the configurable predefined types.

44. A computing device configured to automatically validate results of task performance, comprising:

one or more processors;

a first component that consists of a means for receiving information about a task that is available from a task requester and that has one or more associated validation criteria, the first component being configured to, when executed by at least one of the one or more processors, receive the information about the task that is available from the task requester and that has the one or more associated validation criteria; and

a second component that consists of a means for, after receiving one or more results from performance of the task by one or more task performers, automatically determining whether the received one or more results are validated based at least in part on whether the received one or more results satisfy the one or more associated validation criteria for the task, the second component being configured to, when executed by at least one of the one or more processors, and after the receiving of the one or more results from the performance of the task by the one or more task performers, automatically perform the determining of whether the received one or more results are validated based at least in part on whether the received one or more results satisfy the one or more associated validation criteria for the task.

45. The computing device of claim 44 wherein the second component is further configured to, if it is determined that received results of a task are validated, provide the received results and an indication of the validation of the received results to a task requester who supplied the task.

46. The computing device of claim 44 wherein the one or more associated validation criteria include multiple validation tests, and wherein the determining of whether the received one or more results are validated includes automatically performing multiple of the validation tests.

47. The computing device of claim 44 further comprising a third component that is configured to, after the received one or more results are determined to be validated based at least in part on satisfying the one or more associated validation criteria for the task, provide payment to the one or more task performers based at least in part on the received results being validated.

48. The computing device of claim 44 wherein the first and second components each include instructions for execution in memory of the computing device.