Task-agnostic integration of human and machine intelligence

• US 9,489,636 B2
• Filed: 04/16/2013
• Issued: 11/08/2016
• Est. Priority Date: 04/18/2012
• Status: Active Grant

• Current Assignee: New York University, Tagasauris, Inc.
• Original Assignee: New York University, Tagasauris, Inc.
• Inventors: Attenburg, Joshua M., Ipeirotis, Panagiotis G.
• Primary Examiner(s): Chaki, Kakali
• Assistant Examiner(s): Wu, Fuming
• Application Number:
  US13/863,751
• Publication Number:
  US 20130282630A1
• Time in Patent Office: 1,302 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 5/02 Knowledge representation; Symbolic representation

Abstract

A system combines inputs from human processing and machine processing, and employs machine learning to improve processing of individual tasks based on comparison of human processing results. Once performance of a particular task by machine processing reaches a threshold, the level of human processing used on that task is reduced.

Claims

1. A computer-implemented method comprising:

receiving at least one user action from a user input device, the at least one user action including a plurality of actions performed on an object;

determining a type of object that includes the object on which the at least one user action was performed;

determining a feature vector of the at least one user action and the object type on which the at least one user action was performed;

using the at least one user action and the feature vector to create a set of training data, the training data used to predict future user actions for objects of the type and to determine an accuracy of the predicted future user actions; and

selecting a sample size of users from which to receive additional user actions for the object, the sample size selected responsive to the determined accuracy of the predicted future user actions, the sample size reflecting an extent to which the predicted future user actions substitute for the at least one user action from the user input device, substitution beginning at least in response to the determined accuracy of the predicted user actions being at or near the accuracy of the at least one user action and the extent of substitution increasing over time as the determined accuracy of predicted user actions improves relative to the accuracy of the at least one user action.

2. The method of claim 1, wherein the at least one user action corresponds to more than one classifier performed on a specific object.

3. The method of claim 1, wherein the feature vector is determined using a plurality of feature vector extraction algorithms for the object type.

4. The method of claim 1, wherein the feature vector is determined by including at least two user actions corresponding to at least two related objects, the feature vector determined by analyzing the at least two user actions in aggregate.

5. The computer-implemented method of claim 1, wherein the at least one user action associates keywords with an image.

6. The computer-implemented method of claim 1, further comprising:

extracting information from media objects, wherein the feature vector relates to the information.

7. The computer-implemented method of claim 6, wherein the media object comprises one or more digitally recorded images.

8. The computer-implemented method of claim 6, wherein the media object comprises digitally recorded audio.

9. The computer-implemented method of claim 1, further comprising:

referencing an action library comprising a variety of reference actions; and

using the variety of reference actions to identify the at least one user action.

10. The computer-implemented method of claim 1, further comprising:

identifying a number of the additional actions using cross-validation on the set of training data.

11. A non-transitory computer-readable medium that includes instructions that, when loaded into memory, cause a processor to perform a method, the method comprising:

receiving at least one user action from a user input device, the at least one user action including a plurality of actions performed on an object;

determining a type of object that includes the object on which the at least one user action was performed;

determining a feature vector of the at least one user action and the object type on which the at least one user action was performed;

using the at least one user action and the feature vector to create a set of training data, the training data used to predict future user actions for objects of the type and to determine an accuracy of the predicted future user actions; and

selecting a sample size of users from which to receive additional user actions for the object, the sample size selected responsive to the determined accuracy of the predicted future user actions, the sample size reflecting an extent to which the predicted future user actions substitute for the at least one user action from the user input device, substitution beginning at least in response to the determined accuracy of the predicted user actions being at or near the accuracy of the at least one user action and the extent of substitution increasing over time as the determined accuracy of predicted user actions improves relative to the accuracy of the at least one user action.

12. The non-transitory computer-readable medium of claim 11, wherein the at least one user action corresponds to more than one classifier performed on a specific object.

13. The non-transitory computer-readable medium of claim 11, wherein the feature vector is determined using a plurality of feature vector extraction algorithms for the object type.

14. The non-transitory computer-readable medium of claim 11, wherein the feature vector is determined by including at least two user actions corresponding to at least two related objects, the feature vector determined by analyzing the at least two user actions in aggregate.

15. The non-transitory computer-readable medium of claim 11, wherein the at least one user action associates keywords with an image.

16. The non-transitory computer-readable medium of claim 11, further comprising:

extracting information from media objects, wherein the feature vector relates to the information.

17. The non-transitory computer-readable medium of claim 16, wherein the media object comprises one or more digitally recorded images.

18. The non-transitory computer-readable medium of claim 16, wherein the media object comprises digitally recorded audio.

19. The non-transitory computer-readable medium of claim 11, further comprising:

referencing an action library comprising a variety of reference actions; and

using the variety of reference actions to identify the at least one user action.

20. The non-transitory computer-readable medium of claim 11, further comprising:

identifying a number of the additional actions using cross-validation on the set of training data.