Updateable predictive analytical modeling

• US 8,250,009 B1
• Filed: 09/26/2011
• Issued: 08/21/2012
• Est. Priority Date: 01/26/2011
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Breckenridge, Jordan M., Green, Travis H. K., Kaplow, Robert, Lin, Wei-Hao, Mann, Gideon S.
• Primary Examiner(s): FERNANDEZ RIVAS, OMAR F
• Application Number:
  US13/245,564
• Time in Patent Office: 330 Days
• Field of Search: None
• US Class Current: 706/14
• CPC Class Codes: G06N 20/00 Machine learning

Abstract

Methods, systems, and apparatus, including computer programs encoded on one or more computer storage devices, for training and retraining predictive models. A series of training data sets for predictive modeling can be received, e.g., over a network from a client computing system. The training data included in the training data sets is different from initial training data that was used with multiple training functions to train multiple trained predictive models stored in a predictive model repository. The series of training data sets are used with multiple trained updateable predictive models obtained from the predictive model repository and multiple training functions to generate multiple retrained predictive models. An effectiveness score is generated for each of the retrained predictive models. A first trained predictive model is selected from among the trained predictive models included in the predictive model repository and the retrained predictive models based on their respective effectiveness scores.

Claims

1. A computer-implemented system comprising:

one or more computers;

one or more data storage devices coupled to the one or more computers, storing;

a repository of training functions,

a predictive model repository of trained predictive models, including a plurality of updateable trained predictive models, and wherein each trained predictive model is associated with an effectiveness score that represents an estimation of the effectiveness of the respective trained predictive model, and

instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising;

receiving over a network a series of training data sets for predictive modeling from a client computing system, wherein training data included in the training data sets is different from initial training data that was used with a plurality of training functions obtained from the repository to train the trained predictive models stored in the predictive model repository;

using the series of training data sets, a plurality of trained updateable predictive models obtained from the predictive model repository and a plurality of training functions obtained from the repository of training functions to generate a plurality of retrained predictive models;

generating an effectiveness score for each of the plurality of retrained predictive models;

selecting a first trained predictive model from among the plurality of trained predictive models included in the predictive model repository and the plurality of retrained predictive models based on their respective effectiveness scores; and

providing access to the first trained predictive model over the network.

2. The system of claim 1, wherein the series of training data sets are received incrementally.

3. The system of claim 1, wherein the series of training data sets are received together in a batch.

4. The system of claim 1, wherein the operations further comprise:

for each of the plurality of retrained predictive models;

comparing the effectiveness score of the retrained predictive model to the effectiveness score of the updateable trained predictive model from the predictive model repository that was used to generate the retrained predictive model; and

based on the comparison, selecting a first of the two predictive models to store in the repository of predictive models and not storing a second of the two predictive models in the repository.

5. The system of claim 1, wherein using the series of training data sets to generate the plurality of retrained predictive models occurs in response to determining that a request to update the repository of predictive models has been received from the client computing system.

6. The system of claim 1, wherein using the series of training data sets to generate the plurality of retrained predictive models occurs in response to determining that a size of the training data included in the received series of training data sets has reached or exceeded a threshold size.

7. The system of claim 1, wherein using the series of training data sets to generate the plurality of retrained predictive models occurs in response to determining that a predetermined period of time has expired.

8. The system of claim 1, wherein the operations further comprise:

generating updated training data that includes a least some of the initial training data and at least some of the training data included in the series of training data sets;

generating a second plurality of predictive models using the updated training data and a plurality of training functions obtained from the repository of training functions;

for each of the second plurality of predictive models, generating a respective effectiveness score;

selecting a second trained predictive model based on the effectiveness scores of the second plurality of predictive models; and

providing access to the second trained predictive model over the network.

9. The system of claim 8, wherein selecting a second trained predictive model based on the effectiveness scores of the second plurality of predictive models comprises selecting a second trained predictive model from among the second plurality of predictive models.

10. The system of claim 8, wherein selecting a second trained predictive model based on the effectiveness scores of the second plurality of predictive models comprises selecting a second trained predictive model from among the second plurality of predictive models and the plurality of retrained predictive models and is further based on the effectiveness scores of the retrained predictive models.

11. The system of claim 8, wherein selecting a second trained predictive model based on the effectiveness scores of the second plurality of predictive models comprises selecting a second trained predictive model from among the second plurality of predictive models and the predictive models included in the predictive model repository and is further based on the effectiveness scores of the predictive models included in the predictive model repository.

12. The system of claim 8, wherein generating the second plurality of predictive models occurs in response to determining that a request to update the repository of predictive models has been received from the client computing system.

13. The system of claim 8, wherein generating the second plurality of predictive models occurs in response to determining that a size of the updated training data has reached or exceeded a threshold size.

14. The system of claim 8, wherein generating the second plurality of predictive models occurs in response to determining that a predetermined period of time has expired.

15. The system of claim 1, where the operations further comprise:

receiving input data, data identifying the first trained predictive model, and a request for a predictive output; and

generating the predictive output using the first predictive model and the input data.

16. A computer-implemented method comprising:

receiving over a network a series of training data sets for predictive modeling from a client computing system, wherein training data included in the training data sets is different from initial training data that was used with a plurality of training functions obtained from a repository of training functions to train a plurality of trained predictive models stored in a predictive model repository;

using the series of training data sets, a plurality of trained updateable predictive models obtained from the predictive model repository and a plurality of training functions obtained from the repository of training functions to generate a plurality of retrained predictive models;

generating an effectiveness score for each of the plurality of retrained predictive models;

selecting a first trained predictive model from among the plurality of trained predictive models included in the predictive model repository and the plurality of retrained predictive models based on their respective effectiveness scores; and

providing access to the first trained predictive model over the network.

17. The method of claim 16, wherein using the series of training data sets to generate the plurality of retrained predictive models occurs in response to determining that a size of the training data included in the received series of training data sets has reached or exceeded a threshold size.

18. The method of claim 16, further comprising:

generating updated training data that includes a least some of the initial training data and at least some of the training data included in the series of training data sets;

generating a second plurality of predictive models using the updated training data and a plurality of training functions obtained from the repository of training functions;

for each of the second plurality of predictive models, generating a respective effectiveness score;

selecting a second trained predictive model based on the effectiveness scores of the second plurality of predictive models; and

providing access to the second trained predictive model over the network.

19. The method of claim 18, wherein generating the second plurality of predictive models occurs in response to determining that a predetermined period of time has expired.

20. The method of claim 16, further comprising:

receiving input data, data identifying the first trained predictive model, and a request for a predictive output; and

generating the predictive output using the first predictive model and the input data.

21. A computer-readable storage device encoded with a computer program product, the computer program product comprising instructions that when executed on one or more computers cause the one or more computers to perform operations implementing an adaptable predictive model training system, the operations comprising:

receiving over a network a series of training data sets for predictive modeling from a client computing system, wherein training data included in the training data sets is different from initial training data that was used with a plurality of training functions obtained from a repository of training functions to train a plurality of trained predictive models stored in a predictive model repository;

using the series of training data sets, a plurality of trained updateable predictive models obtained from the predictive model repository and a plurality of training functions obtained from the repository of training functions to generate a plurality of retrained predictive models;

generating an effectiveness score for each of the plurality of retrained predictive models;

selecting a first trained predictive model from among the plurality of trained predictive models included in the predictive model repository and the plurality of retrained predictive models based on their respective effectiveness scores; and

providing access to the first trained predictive model over the network.

22. The computer-readable storage device of claim 21, wherein using the series of training data sets to generate the plurality of retrained predictive models occurs in response to determining that a size of the training data included in the received series of training data sets has reached or exceeded a threshold size.

23. The computer-readable storage device of claim 21, the operations further comprising:

generating updated training data that includes a least some of the initial training data and at least some of the training data included in the series of training data sets;

generating a second plurality of predictive models using the updated training data and a plurality of training functions obtained from the repository of training functions;

for each of the second plurality of predictive models, generating a respective effectiveness score;

selecting a second trained predictive model based on the effectiveness scores of the second plurality of predictive models; and

providing access to the second trained predictive model over the network.

24. The computer-readable storage device of claim 23, wherein generating the second plurality of predictive models occurs in response to determining that a predetermined period of time has expired.

25. The computer-readable storage device of claim 21, the operations further comprising:

receiving input data, data identifying the first trained predictive model, and a request for a predictive output; and

generating the predictive output using the first predictive model and the input data.