Assessing accuracy of trained predictive models

• US 8,533,224 B2
• Filed: 05/04/2011
• Issued: 09/10/2013
• Est. Priority Date: 05/04/2011
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Lin, Wei-Hao, Green, Travis, Kaplow, Robert, Fu, Gang, Mann, Gideon S.
• Primary Examiner(s): SYED, FARHAN M
• Application Number:
  US13/101,048
• Publication Number:
  US 20120284213A1
• Time in Patent Office: 860 Days
• US Class Current: 707/777
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 3/08 Learning methods

Abstract

A system includes a computer(s) coupled to a data storage device(s) that stores a training data repository and a predictive model repository. The training data repository includes retained data samples from initial training data and from previously received data sets. The predictive model repository includes at least one updateable trained predictive model that was trained with the initial training data and retrained with the previously received data sets. A new data set is received. A richness score is assigned to each of the data samples in the set and to the retained data samples that indicates how information rich a data sample is for determining accuracy of the trained predictive model. A set of test data is selected based on ranking by richness score the retained data samples and the new data set. The trained predictive model is accuracy tested using the test data and an accuracy score determined.

Claims

1. A computer-implemented method comprising:

receiving a first data set of data samples, each data sample comprising input data and corresponding output data, wherein the first data set is new relative to (i) an initial training data set and (ii) a plurality of previously received update data sets of data samples, wherein the initial training data set was used to train each trained predictive model in a repository of trained predictive models, at least some which are updateable, and wherein the plurality of previously received update data sets of the data sample were used to retrain one or more updateable trained predictive models in the repository;

assigning a richness score to each of the data samples included in the first data set and to each of a set of retained data samples from the initial training data and the plurality of previously received update data sets, wherein the richness score for a particular data sample indicates how information rich the particular data sample is, relative to other data samples in the set of retained data samples and the first data set, for determining an accuracy of a trained predictive model;

ranking the data samples included in the first data set and the set of retained data samples based on the assigned richness scores;

selecting a first set of test data from the data samples included in the first data set and the set of retained data samples based on the ranking;

testing how accurate each of the trained predictive models in the repository is in determining predictive output data for given input data using the first set of test data and determining respective accuracy scores for each of the trained predictive models based on the testing; and

selecting a first trained predictive model from the repository based on the accuracy scores and providing access to the first trained predictive model to a client computing system for generating predictive output data based on input data received from the client computing system.

2. The method of claim 1, further comprising:

after determining accuracy scores for each of the trained predictive models, retraining each of the updateable trained predictive models included in the repository using the first data set; and

updating the repository to replace the updateable trained predictive models with the retrained predictive models, where each retrained predictive model is associated with the accuracy score determined for the trained predictive model from which the retrained predictive model was derived.

3. The method of claim 1, wherein assigning a richness score to the particular data sample comprises determining the richness score based on how many data samples have similar input data but different output data than the particular data sample and based on how many data samples have similar input data and similar or different output data than the particular data sample.

4. The method of claim 1, wherein selecting the first set of test data from the data samples comprises selecting the top n^th ranked data samples where n is an integer greater than one.

5. The method of claim 1, wherein:

testing how accurate the trained predictive model is in determining predictive output data for given input data using the first set of test data comprises generating predictive output data for the input data included in the data samples of the first set of test data; and

determining an accuracy score based on the testing comprises comparing the predictive output data to the output data included the data samples that correspond to the input data used to generate the predictive output data and determining the accuracy score based on the comparison.

6. The method of claim 1, further comprising:

after determining the accuracy score for the trained predictive model, retraining the trained predictive model using the first data set of data samples.

7. A computer-implemented system comprising:

one or more computers; and

one or more data storage devices coupled to the one or more computers, storing;

a training data repository that includes a set of retained data samples, wherein the set of retained data samples includes at least some data samples from an initial training data set and some data samples from a plurality of previously received update data sets, wherein each data sample includes input data and corresponding output data;

a predictive model repository that includes trained predictive models that were each trained with the initial training data set, wherein at least some of the trained predictive models are updateable and were each retrained with the plurality of previously received update data sets, and

instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising;

receiving a first data set of data samples, each data sample comprising input data and corresponding output data, wherein the first data set is new relative to (i) the initial training data set and (ii) the plurality of previously received update data sets;

assigning a richness score to each of the data samples included in the first data set and to each of the set of retained data samples included in the training data repository, wherein the richness score for a particular data sample indicates how information rich the particular data sample is, relative to other data samples in the set of retained data samples and the first data set, for determining an accuracy of a trained predictive model;

ranking the data samples included in the first data set and the retained data samples based on the assigned richness scores;

selecting a first set of test data from the data samples included in the first data set and the set of retained data samples based on the ranking;

testing how accurate each of the trained predictive models in the repository is in determining predictive output data for given input data using the first set of test data and determining respective accuracy scores for each of the trained predictive models based on the testing; and

selecting a first trained predictive model from the repository based on the accuracy scores and providing access to the first trained predictive model to a client computing system for generating predictive output data based on input data received from the client computing system.

8. The system of claim 7, wherein the operations further comprise:

after determining accuracy scores for each of the trained predictive models, retraining each of the updateable trained predictive models included in the repository using the first data set; and

updating the repository to replace the updateable trained predictive models with the retrained predictive models, where each retrained predictive model is associated with the accuracy score determined for the trained predictive model from which the retrained predictive model was derived.

9. The system of claim 7, wherein assigning a richness score to the particular data sample comprises determining the richness score based on how many data samples have similar input data but different output data than the particular data sample and based on how many data samples have similar input data and similar or different output data than the particular data sample.

10. The system of claim 7, wherein selecting the first of test data from the data samples comprises selecting the top n^th ranked data samples where n is an integer greater than one.

11. The system of claim 7, wherein:

testing how accurate the trained predictive model is in determining predictive output data for given input data using the first set of test data comprises generating predictive output data for the input data included in the data samples of the first set of test data; and

determining an accuracy score based on the testing comprises comparing the predictive output data to the output data included the data samples that correspond to the input data used to generate the predictive output data and determining the accuracy score based on the comparison.

12. The system of claim 7, further comprising:

after determining the accuracy score for the trained predictive model, retraining the trained predictive model using the first data set of data samples.

13. A computer-readable storage device encoded with a computer program product, the computer program product comprising instructions that when executed on one or more computers cause the one or more computers to perform operations comprising:

receiving a first data set of data samples, each data sample comprising input data and corresponding output data, wherein the first data set is new relative to (i) an initial training data set and (ii) a plurality of previously received update data sets of data samples, wherein the initial training data set was used to train each trained predictive model in a repository of trained predictive models, at least some which are updateable, and wherein the plurality of previously received update data sets of the data sample were used to retrain one or more updateable trained predictive models in the repository;

assigning a richness score to each of the data samples included in the first data set and to each of a set of retained data samples from the initial training data and the plurality of previously received update data sets, wherein the richness score for a particular data sample indicates how information rich the particular data sample is, relative to other data samples in the set of retained data samples and the first data set, for determining an accuracy of a trained predictive model;

ranking the data samples included in the first data set and the set of retained data samples based on the assigned richness scores;

selecting a first set of test data from the data samples included in the first data set and the set of retained data samples based on the ranking;

testing how accurate each of the trained predictive models in the repository is in determining predictive output data for given input data using the first set of test data and determining respective accuracy scores for each of the trained predictive models based on the testing; and

selecting a first trained predictive model from the repository based on the accuracy scores and providing access to the first trained predictive model to a client computing system for generating predictive output data based on input data received from the client computing system.

14. The computer-readable storage device of claim 13, the operations further comprising:

after determining accuracy scores for each of the trained predictive models, retraining each of the updateable trained predictive models included in the repository using the first data set; and

updating the repository to replace the updateable trained predictive models with the retrained predictive models, where each retrained predictive model is associated with the accuracy score determined for the trained predictive model from which the retrained predictive model was derived.

15. The computer-readable storage device of claim 13, wherein assigning a richness score to the particular data sample comprises determining the richness score based on how many data samples have similar input data but different output data than the particular data sample and based on how many data samples have similar input data and similar or different output data than the particular data sample.

16. The computer-readable storage device of claim 13, wherein selecting the first set of test data from the data samples comprises selecting the top n^th ranked data samples where n is an integer greater than one.

17. The computer-readable storage device of claim 13, wherein:

testing how accurate the trained predictive model is in determining predictive output data for given input data using the first set of test data comprises generating predictive output data for the input data included in the data samples of the first set of test data; and

determining an accuracy score based on the testing comprises comparing the predictive output data to the output data included the data samples that correspond to the input data used to generate the predictive output data and determining the accuracy score based on the comparison.

18. The computer-readable storage device of claim 13, the operations further comprising:

after determining the accuracy score for the trained predictive model, retraining the trained predictive model using the first data set of data samples.