Systems and methods for selecting machine learning training data

• US 10,325,224 B1
• Filed: 07/07/2017
• Issued: 06/18/2019
• Est. Priority Date: 03/23/2017
• Status: Active Grant

• Current Assignee: Palantir Technologies Incorporated
• Original Assignee: Palantir Technologies Incorporated
• Inventors: Erenrich, Daniel, Elkherj, Matthew
• Primary Examiner(s): Wong, Lut
• Application Number:
  US15/644,231
• Time in Patent Office: 711 Days
• Field of Search: 706 45
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 5/04 Inference or reasoning models

Abstract

Systems and methods are provided for selecting training examples to increase the efficiency of supervised active machine learning processes. Training examples for presentation to a user may be selected according to measure of the model'"'"'s uncertainty in labeling the examples. A number of training examples may be selected to increase efficiency between the user and the processing system by selecting the number of training examples to minimize user downtime in the machine learning process.

Claims

1. An entity resolution system utilizing active learning for training a machine learning model of the entity resolution system, the entity resolution system comprising:

one or more processors and a memory storing instructions that, when executed by the one or more processors, cause the system to;

obtain a machine learning model and a training dataset, the training dataset including a plurality of training examples, each training example of at least a portion of the training examples including one or more records, each record including an entity identification field and an entity location field;

determine uncertainty scores for the plurality of training examples according to the machine learning model;

select a first example batch from the plurality of training examples according to uncertainty scores of the plurality of training examples;

update the machine learning model according to at least one labeled training example of the first example batch;

determine updated uncertainty scores for the plurality of training examples according to the updated machine learning model;

select a second example batch from the plurality of training examples according to the updated uncertainty scores of the plurality of training examples;

update the machine learning model according to at least one labeled training example of the second example batch;

resolving, based at least in part on the machine learning model updated according to the at least one labeled training example of the second example batch, matching entities associated with one or more sets of sets of records, the one or more sets of records including at least a portion of the training dataset.

2. The system for training a machine learning model of claim 1, wherein to update the machine learning model the system is further caused to:

obtain a first label of a first training example of the first example batch from a user, wherein the first training example is selected according to the uncertainty scores of the first example batch;

update the machine learning model a first time according to the first label;

determine updated uncertainty scores of the first example batch;

obtain a second label of a second training example of the first example batch from the user, wherein the second training example is selected according to the updated uncertainty scores of the first example batch; and

update the machine learning model a second time according to the second label.

3. The system for training a machine learning model of claim 1, wherein to determine updated uncertainty scores for the plurality of training examples, the system is further caused to:

determine updated uncertainty scores for the plurality of training examples according to the updated model after a threshold number of training examples of the first example batch are labeled by a user.

4. The system for training a machine learning model of claim 3, wherein the threshold number is determined according to at least one of:

a number of training examples in the first example batch, an estimated time for a user to label a training example, a system processing speed, or a number of training examples of the training dataset.

5. The system for training a machine learning model of claim 4, wherein the threshold number is determined such that a time to completion of labeling the threshold number of training examples of the first batch and a time to completion of determining updated uncertainty scores of the plurality of training examples of the training dataset are within a predetermined range of each other.

6. The system for training a machine learning model of claim 1, wherein the system is further caused to determine a machine learning model certainty level according to the updated uncertainty scores of the plurality of training examples of the training dataset.

7. The system for training a machine learning model of claim 6, wherein to select the second example batch the system is further caused to determine to select the second example batch according to the machine learning model certainty level.

8. The system for training a machine learning model of claim 1, wherein the system is further caused to determine a number of training examples in the second example batch.

9. The system for training a machine learning model of claim 8, wherein to determine the number of training examples in the second example batch, the system is further caused to determine a machine learning model certainty level according to the updated uncertainty scores of the plurality of training examples of the training dataset.

10. The system for training a machine learning model of claim 8, wherein to determine the number of training examples in the second example batch, the system is further caused to compare an estimated time for a user to label a training example and an estimated time for the system to update uncertainty scores of the training examples of the second example batch.

11. A method for entity resolution utilizing active learning for training a machine learning model, the method being performed on a computer system having one or more physical processors programmed with computer program instructions that, when executed by the one or more physical processors, cause the computer system to perform the method, the method comprising:

obtaining, by the computer system, a machine learning model and a training dataset, the training dataset including a plurality of training examples, each training example of at least a portion of the training examples including one or more records, each record including an entity identification field and an entity location field;

determining, by the computer system, uncertainty scores for the plurality of training examples according to the machine learning model;

selecting, by the computer system, a first example batch from the plurality of training examples according to uncertainty scores of the plurality of training examples;

updating, by the computer system, the machine learning model according to at least one labeled training example of the first example batch;

determining, by the computer system, updated uncertainty scores for the plurality of training examples according to the updated machine learning model;

selecting, by the computer system, a second example batch from the plurality of training examples according to the updated uncertainty scores of the plurality of training examples;

updating, by the computer system, the machine learning model according to at least one labeled training example of the second example batch;

resolving, by the computer system based at least in part on the machine learning model updated according to the at least one labeled training example of the second example batch, matching entities associated with one or more sets of sets of records, the one or more sets of records including at least a portion of the training dataset.

12. The method of claim 11, wherein updating the machine learning model further includes:

obtaining, by the computer system, a first label of a first training example of the first example batch from a user, wherein the first training example is selected according to the uncertainty scores of the first example batch;

updating, by the computer system, the machine learning model a first time according to the first label;

determining, by the computer system, updated uncertainty scores of the first example batch;

obtaining, by the computer system, a second label of a second training example of the first example batch from the user, wherein the second training example is selected according to the updated uncertainty scores of the first example batch; and

updating, by the computer system, the machine learning model a second time according to the second label.

13. The method of claim 11, wherein determining updated uncertainty scores for the plurality of training examples further includes:

determining, by the computer system, updated uncertainty scores for the plurality of training examples according to the updated model after a threshold number of training examples of the first example batch are labeled by a user.

14. The method of claim 13, wherein the threshold number is determined according at least one of:

a number of training examples in the first example batch, an estimated time for a user to label a training example, a system processing speed, or a number of training examples of the training dataset.

15. The method of claim 14, wherein the threshold number is determined such that a time to completion of labeling the threshold number of training examples of the first batch and a time to completion of determining updated uncertainty scores of the plurality of training examples of the training dataset are within a predetermined range of each other.

16. The method of claim 11, further comprising determining, by the computer system, a machine learning model certainty level according to the updated uncertainty scores of the plurality of training examples of the training dataset.

17. The method of claim 16, wherein selecting the second example batch the system further includes determining, by the computer system, to select the second example batch according to the machine learning model certainty level.

18. The method of claim 11, further comprising determining, by the computer system, a number of training examples in the second example batch.

19. The method of claim 18, wherein determining the number of training examples in the second example batch further includes determining, by the computer system, a machine learning model certainty level according to the updated uncertainty scores of the plurality of training examples of the training dataset.

20. The method of claim 18, wherein determining the number of training examples in the second example batch further includes comparing, by the computer system, an estimated time for a user to label a training example and an estimated time for the system to update uncertainty scores of the training examples of the second example batch.