Task-agnostic Integration of Human and Machine Intelligence

• US 20130282630A1
• Filed: 04/16/2013
• Published: 10/24/2013
• Est. Priority Date: 04/18/2012
• Status: Active Grant

• Current Assignee: New York University, Tagasauris, Inc.
• Original Assignee: Tagasauris, Inc.
• Inventors: Ipeirotis, Panagiotis G., Attenberg, Joshua M.
• Granted Patent: US 9,489,636 B2
• Time in Patent Office: Days
• US Class Current: 706/12
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 5/02 Knowledge representation; Symbolic representation

Abstract

A system combines inputs from human processing and machine processing, and employs machine learning to improve processing of individual tasks based on comparison of human processing results. Once performance of a particular task by machine processing reaches a threshold, the level of human processing used on that task is reduced.

Claims

1. A computer-implemented method comprising:

receiving at least one user action from a user input device, the at least one user action including a plurality of actions performed on an object;

determining a type of object that includes the object on which the at least one user action was performed;

determining a feature vector of the at least one user action and the object type on which the at least one user action was performed;

using at least one user action and the feature vector to create a set of training data, the training data used to predict future user actions for objects of the type and to determine an accuracy of the predicted future user actions; and

selecting a sample size of users from which to receive additional user actions for the object, the sample size selected responsive to the determined accuracy of the predicted future user actions.

2. The method of claim 1, wherein the at least one user action corresponds to more than one classifier performed on a specific object.

3. The method of claim 1, wherein the feature vector is determined using a plurality of feature vector extraction algorithms for the object type.

4. The method of claim 1, wherein the feature vector is determined by including at least two user actions corresponding to at least two related objects, the feature vector determined by analyzing the at least two user actions in aggregate.

5. A non-transitory computer-readable medium that includes instructions that, when loaded into memory, cause a processor to perform a method, the method comprising:

receiving at least one user action from a user input device, the at least one user action including a plurality of actions performed on an object;

determining a type of object that includes the object on which the at least one user action was performed;

determining a feature vector of the at least one user action and the object type on which the at least one user action was performed;

using at least one user action and the feature vector to create a set of training data, the training data used to predict future user actions for objects of the type and to determine an accuracy of the predicted future user actions; and

selecting a sample size of users from which to receive additional user actions for the object, the sample size selected responsive to the determined accuracy of the predicted future user actions.

6. The non-transitory computer-readable medium of claim 5, wherein the at least one user action corresponds to more than one classifier performed on a specific object.

7. The non-transitory computer-readable medium of claim 5, wherein the feature vector is determined using a plurality of feature vector extraction algorithms for the object type.

8. The non-transitory computer-readable medium of claim 5, wherein the feature vector is determined by including at least two user actions corresponding to at least two related objects, the feature vector determined by analyzing the at least two user actions in aggregate.

9. A computer program product for a machine learning system, wherein the computer program product is stored on a computer-readable medium that includes instructions that, when loaded into memory, cause a processor to perform a method, the method comprising:

receiving at least one user action from a user input device, the at least one user action including a plurality of actions performed on an object;

determining a type of object that includes the object on which the at least one user action was performed;

determining a feature vector of the at least one user action and the object type on which the at least one user action was performed;

using at least one user action and the feature vector to create a set of training data, the training data used to predict future user actions for objects of the type and to determine an accuracy of the predicted future user actions; and

selecting a sample size of users from which to receive additional user actions for the object, the sample size selected responsive to the determined accuracy of the predicted future user actions.

10. The computer program product of claim 9, wherein the at least one user action corresponds to more than one classifier performed on a specific object.

11. The computer program product of claim 9, wherein the feature vector is determined using a plurality of feature vector extraction algorithms for the object type.

12. The computer program product of claim 9, wherein the feature vector is determined by including at least two user actions corresponding to at least two related objects, the feature vector determined by analyzing the at least two user actions in aggregate.