Predictive Analytical Modeling Data Selection

• US 20120284213A1
• Filed: 05/04/2011
• Published: 11/08/2012
• Est. Priority Date: 05/04/2011
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Lin, Wei-Hao, Green, Travis, Kaplow, Robert, Fu, Gang, Mann, Gideon S.
• Granted Patent: US 8,533,224 B2
• Time in Patent Office: Days
• US Class Current: 706/12
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 3/08 Learning methods

Abstract

A system includes a computer(s) coupled to a data storage device(s) that stores a training data repository and a predictive model repository. The training data repository includes retained data samples from initial training data and from previously received data sets. The predictive model repository includes at least one updateable trained predictive model that was trained with the initial training data and retrained with the previously received data sets. A new data set is received. A richness score is assigned to each of the data samples in the set and to the retained data samples that indicates how information rich a data sample is for determining accuracy of the trained predictive model. A set of test data is selected based on ranking by richness score the retained data samples and the new data set. The trained predictive model is accuracy tested using the test data and an accuracy score determined.

Claims

1. A computer-implemented method comprising:

receiving a new data set of data samples, each data sample comprising input data and corresponding output data, wherein the data set is new compared to an initial training data set and a plurality of previously received update data sets of data samples that were used to train and retrain respectively an updateable trained predictive model;

assigning a richness score to each of the data samples included in the new data set and to retained data samples from the initial training data and the plurality of previously received data sets, wherein the richness score for a particular data sample indicates how information rich the particular data sample is relative to other retained data samples for determining an accuracy of the trained predictive model;

ranking the data samples included in the new data set and the retained data samples based on the assigned richness scores;

selecting a set of test data from the data samples included in the new data set and the retained data samples based on the ranking; and

testing how accurate the trained predictive model is in determining predictive output data for given input data using the set of test data and determining an accuracy score for the trained predictive model based on the testing.

2. The method of claim 1, wherein the trained predictive model is included in a repository of trained predictive models that were all trained using the same initial training data set and at least some of which are updateable and were retrained using the plurality of received update data sets, the method further comprising:

testing how accurate each of the trained predictive models in the repository is using the set of test data and determining accuracy scores based on the testing for each of the trained predictive models; and

selecting a first trained predictive model from the repository of trained predictive models based on the accuracy scores and providing access to the first trained predictive model to a client computing system for generating predictive output data based on input data received from the client computing system.

3. The method of claim 2, further comprising:

after determining accuracy scores for each of the trained predictive models, retraining each of the updateable trained predictive models included in the repository using the new data set; and

updating the repository to replace the updateable trained predictive models with the retrained predictive models, where each retrained predictive model is associated with the accuracy score determined for the trained predictive model from which the retrained predictive model was derived.

4. The method of claim 1, wherein assigning a richness score to the particular data sample comprises determining the richness score based on how many data samples have similar input data but different output data than the particular data sample and based on how many data samples have similar input data and similar or different output data than the particular data sample.

5. The method of claim 1, wherein selecting a set of test data from the data samples comprises selecting the top n^th ranked data samples where n is an integer greater than one.

6. The method of claim 1, wherein:

testing how accurate the trained predictive model is in determining predictive output data for given input data using the set of test data comprises generating predictive output data for the input data included in the data samples of the test data set; and

determining an accuracy score based on the testing comprises comparing the predictive output data to the output data included the data samples that correspond to the input data used to generate the predictive output data and determining the accuracy score based on the comparison.

7. The method of claim 1, further comprising;

after determining the accuracy score for the trained predictive model, retraining the trained predictive model using the new data set of data samples.

8. A computer-implemented system comprising:

one or more computers; and

one or more data storage devices coupled to the one or more computers, storing;

a training data repository of retained data samples that includes at least some data samples from an initial training data set and from a plurality of previously received update data sets, wherein each data sample includes input data and corresponding output data;

a predictive model repository that includes at least one updateable trained predictive model that was trained with the initial training data set and retrained with the plurality of previously received update data sets, and

instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising;

receiving a new data set of data samples, each data sample comprising input data and corresponding output data, wherein the data set is new compared to the initial training data set and to the plurality of previously received update data sets;

assigning a richness score to each of the data samples included in the new data set and to the retained data samples included in the training data repository, wherein the richness score for a particular data sample indicates how information rich the particular data sample is relative to other retained data samples for determining an accuracy of the trained predictive model;

ranking the data samples included in the new data set and the retained data samples based on the assigned richness scores;

selecting a set of test data from the data samples included in the new data set and the retained data samples based on the ranking; and

testing how accurate the at least one trained predictive model is in determining predictive output data for given input data using the set of test data and determining an accuracy score for the trained predictive model based on the testing.

9. The system of claim 8, wherein predictive model repository includes a plurality of trained predictive models that were all trained using the same initial training data set and at least some of which are updateable and were retrained using the plurality of received update data sets, and wherein the operations further comprise:

testing how accurate each of the trained predictive models in the repository is using the set of test data and determining accuracy scores based on the testing for each of the trained predictive models; and

selecting a first trained predictive model from the repository of trained predictive models based on the accuracy scores and providing access to the first trained predictive model to a client computing system for generating predictive output data based on input data received from the client computing system.

10. The system of claim 9, wherein the operations further comprise:

after determining accuracy scores for each of the trained predictive models, retraining each of the updateable trained predictive models included in the repository using the new data set; and

updating the repository to replace the updateable trained predictive models with the retrained predictive models, where each retrained predictive model is associated with the accuracy score determined for the trained predictive model from which the retrained predictive model was derived.

11. The system of claim 8, wherein assigning a richness score to the particular data sample comprises determining the richness score based on how many data samples have similar input data but different output data than the particular data sample and based on how many data samples have similar input data and similar or different output data than the particular data sample.

12. The system of claim 8, wherein selecting a set of test data from the data samples comprises selecting the top n^th ranked data samples where n is an integer greater than one.

13. The system of claim 8, wherein:

testing how accurate the trained predictive model is in determining predictive output data for given input data using the set of test data comprises generating predictive output data for the input data included in the data samples of the test data set; and

determining an accuracy score based on the testing comprises comparing the predictive output data to the output data included the data samples that correspond to the input data used to generate the predictive output data and determining the accuracy score based on the comparison.

14. The system of claim 8, further comprising;

after determining the accuracy score for the trained predictive model, retraining the trained predictive model using the new data set of data samples.

15. A computer-readable storage device encoded with a computer program product, the computer program product comprising instructions that when executed on one or more computers cause the one or more computers to perform operations comprising:

receiving a new data set of data samples, each data sample comprising input data and corresponding output data, wherein the data set is new compared to an initial training data set and a plurality of previously received update data sets of data samples that were used to train and retrain respectively an updateable trained predictive model;

assigning a richness score to each of the data samples included in the new data set and to retained data samples from the initial training data and the plurality of previously received data sets, wherein the richness score for a particular data sample indicates how information rich the particular data sample is relative to other retained data samples for determining an accuracy of the trained predictive model;

ranking the data samples included in the new data set and the retained data samples based on the assigned richness scores;

selecting a set of test data from the data samples included in the new data set and the retained data samples based on the ranking; and

testing how accurate the trained predictive model is in determining predictive output data for given input data using the set of test data and determining an accuracy score for the trained predictive model based on the testing.

16. The computer-readable storage device of claim 15, wherein the trained predictive model is included in a repository of trained predictive models that were all trained using the same initial training data set and at least some of which are updateable and were retrained using the plurality of received update data sets, the operations further comprising:

testing how accurate each of the trained predictive models in the repository is using the set of test data and determining accuracy scores based on the testing for each of the trained predictive models; and

selecting a first trained predictive model from the repository of trained predictive models based on the accuracy scores and providing access to the first trained predictive model to a client computing system for generating predictive output data based on input data received from the client computing system.

17. The computer-readable storage device of claim 16, the operations further comprising:

after determining accuracy scores for each of the trained predictive models, retraining each of the updateable trained predictive models included in the repository using the new data set; and

updating the repository to replace the updateable trained predictive models with the retrained predictive models, where each retrained predictive model is associated with the accuracy score determined for the trained predictive model from which the retrained predictive model was derived.

18. The computer-readable storage device of claim 15, wherein assigning a richness score to the particular data sample comprises determining the richness score based on how many data samples have similar input data but different output data than the particular data sample and based on how many data samples have similar input data and similar or different output data than the particular data sample.

19. The computer-readable storage device of claim 15, wherein selecting a set of test data from the data samples comprises selecting the top n^th ranked data samples where n is an integer greater than one.

20. The computer-readable storage device of claim 15, wherein:

testing how accurate the trained predictive model is in determining predictive output data for given input data using the set of test data comprises generating predictive output data for the input data included in the data samples of the test data set; and

determining an accuracy score based on the testing comprises comparing the predictive output data to the output data included the data samples that correspond to the input data used to generate the predictive output data and determining the accuracy score based on the comparison.

21. The computer-readable storage device of claim 15, the operations further comprising;

after determining the accuracy score for the trained predictive model, retraining the trained predictive model using the new data set of data samples.