Updateable predictive analytical modeling

• US 8,533,222 B2
• Filed: 01/26/2011
• Issued: 09/10/2013
• Est. Priority Date: 01/26/2011
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Breckenridge, Jordan M., Green, Travis, Kaplow, Robert, Lin, Wei-Hao, Mann, Gideon S.
• Primary Examiner(s): SYED, FARHAN M
• Application Number:
  US13/014,223
• Publication Number:
  US 20120191630A1
• Time in Patent Office: 958 Days
• US Class Current: 707/774
• CPC Class Codes: G06N 20/00 Machine learning

Abstract

Methods, systems, and apparatus, including computer programs encoded on one or more computer storage devices, for training and retraining predictive models. A series of training data sets for predictive modeling can be received, e.g., over a network from a client computing system. The training data included in the training data sets is different from initial training data that was used with multiple training functions to train multiple trained predictive models stored in a predictive model repository. The series of training data sets are used with multiple trained updateable predictive models obtained from the predictive model repository and multiple training functions to generate multiple retrained predictive models. An effectiveness score is generated for each of the retrained predictive models. A first trained predictive model is selected from among the trained predictive models included in the predictive model repository and the retrained predictive models based on their respective effectiveness scores.

Claims

1. A computer-implemented system comprising:

one or more computers;

one or more data storage devices in data communication with the one or more computers, storing;

a training data repository that includes client training data comprising a first plurality of training data sets belonging to a client entity and received over a network;

a plurality of training functions; and

instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising;

generating a plurality of trained predictive models using the plurality of training functions and a first sample of the client training data;

determining a respective accuracy of each of the plurality of trained predictive models using a different, second sample of the client training data;

receiving, over the network one or more new training data sets belonging to the client entity, wherein each of the one or more new training data sets is new relative to the first plurality of training data sets;

updating the client training data to include the one or more new training data sets;

generating a plurality of new trained predictive models using the plurality of training functions and a different, third sample of the client training data;

determining, a respective accuracy of each of the plurality of new trained predictive models using a different, fourth sample of the client training data;

generating a respective effectiveness score for each of the plurality of trained predictive models and each of the plurality of new trained predictive models using the determined accuracy of its respective trained predictive model;

receiving, over the network from a client computing system, a first prediction request and first input data;

selecting a first trained predictive model to service the first prediction request from among the plurality of trained predictive models and the plurality of new trained predictive models based on the respective effectiveness scores;

running the first trained predictive model on the first input data to generate a predictive output; and

providing, to the client computing system, the predictive output in response to the first prediction request.

2. The computer-implemented system of claim 1, wherein the plurality of trained predictive models includes one or more trained updateable predictive models, and wherein the operations further comprise:

generating a retrained updateable predictive model using a previously-trained updateable predictive model, the training function that was used to generate the previously-trained updateable predictive model, and a different, fifth, sample of the client training data;

determining an accuracy of the retrained updateable predictive model using a different, sixth sample of data of the client training data;

generating an effectiveness score for the retrained updateable predictive model using the determined accuracy of the retrained updateable predictive model;

receiving a second prediction request and second input data;

selecting a second trained predictive model to service the second prediction request from among the retrained updateable predictive model and the plurality of trained predictive models based on the respective effectiveness scores; and

running the second trained predictive model on the second input data in response to the second prediction request.

3. The computer-implemented system of claim 2, wherein the operations further comprise:

determining, before generating the retrained updateable predictive model, that at least one of the following conditions is true;

(i) an amount of training data in a training data queue is greater than or equal to a threshold amount;

(ii) a predetermined amount of time is reached or exceeded;

or (iii) a request to update the previously-trained updateable predictive model is received.

4. The computer-implemented system of claim 1, wherein the operations further comprise:

determining, before generating the plurality of new trained predictive models, that at least one of the following conditions is true;

(i) a predetermined amount of time is reached or exceeded;

or (ii) a request to generate new trained predictive models is received.

5. The computer-implemented system of claim 1, wherein the third sample of client training data includes one or more of the new training data sets and one or more training data sets from the first plurality of training data sets.

6. The computer-implemented system of claim 1, wherein the third sample of client training data does not include any of the training data sets included in the first plurality of training data sets.

7. The computer-implemented system of claim 1, wherein the operations further comprise:

maintaining the training data repository according to a data retention policy that defines rules determining which training data to retain and which training data to delete from the repository based on one or more of the following;

respective dates of receipts of the training data and respective properties of the training data.

20. The computer-implemented system of claim 1, wherein the operations further comprise:

determining a respective resource usage for running each of the plurality of trained predictive models; and

generating a respective effectiveness score for each of the plurality of trained predictive models using its respective determined resource usage.

8. A computer-implemented method comprising:

receiving, over a network, client training data comprising a first plurality of training data sets belonging to a client entity;

generating a plurality of trained predictive models using a plurality of training functions and a first sample of the client training data;

determining a respective accuracy of each of the plurality of trained predictive models using a different, second sample of the client training data;

receiving, over the network one or more new training data sets belonging to the client entity, wherein each of the one or more new training data sets is new relative to the first plurality of training data sets;

updating the client training data to include the one or more new training data sets;

generating a plurality of new trained predictive models using the plurality of training functions and a different, third sample of the client training data;

determining, a respective accuracy of each of the plurality of new trained predictive models using a different, fourth sample of the client training data;

generating a respective effectiveness score for each of the plurality of trained predictive models and each of the plurality of new trained predictive models using the determined accuracy of its respective trained predictive model;

receiving, over the network from a client computing system, a first prediction request and first input data;

selecting a first trained predictive model to service the first prediction request from among the plurality of trained predictive models and the plurality of new trained predictive models based on the respective effectiveness scores;

running the first trained predictive model on the first input data to generate a predictive output; and

providing, to the client computing system, the predictive output in response to the first prediction request.

9. The computer-implemented method of claim 8, wherein the plurality of trained predictive models includes one or more trained updateable predictive models, and wherein the operations further comprise:

generating a retrained updateable predictive model using a previously-trained updateable predictive model, the training function that was used to generate the previously-trained updateable predictive model, and a different, fifth, sample of the client training data;

determining an accuracy of the retrained updateable predictive model using a different, sixth sample of data of the client training data;

generating an effectiveness score for the retrained updateable predictive model using the determined accuracy of the retrained updateable predictive model;

receiving a second prediction request and second input data;

selecting a second trained predictive model to service the second prediction request from among the retrained updateable predictive model and the plurality of trained predictive models based on the respective effectiveness scores; and

running the second trained predictive model on the second input data in response to the second prediction request.

10. The computer-implemented method of claim 9, further comprising:

determining, before generating the retrained updateable predictive model, that at least one of the following conditions is true;

(i) an amount of training data in a training data queue is greater than or equal to a threshold amount;

(ii) a predetermined amount of time is reached or exceeded;

or (iii) a request to update the previously-trained updateable predictive model is received.

11. The computer-implemented method of claim 8, further comprising:

determining, before generating the plurality of new trained predictive models, that at least one of the following conditions is true;

(i) a predetermined amount of time is reached or exceeded;

or (ii) a request to generate new trained predictive models is received.

12. The computer-implemented method of claim 8, wherein the third sample of client training data includes one or more new training data sets and one or more training data sets from the first plurality of training data sets.

13. The computer-implemented method of claim 8, wherein the third sample of client training data does not include any of the training data sets included in the first plurality of training data sets.

21. The computer-implemented method of claim 8, wherein the method further comprises:

determining a respective resource usage for running each of the plurality of trained predictive models; and

generating a respective effectiveness score for each of the plurality of trained predictive models using its respective determined resource usage.

14. A computer-readable storage device encoded with a computer program product, the computer program product comprising instructions that when executed on one or more computers cause the one or more computers to perform operations comprising:

receiving, over a network, client training data comprising a first plurality of training data sets belonging to a client entity;

generating a plurality of trained predictive models using a plurality of training functions and a first sample of the client training data;

determining a respective accuracy of each of the plurality of trained predictive models using a different, second sample of the client training data;

receiving, over the network one or more new training data sets belonging to the client entity, wherein each of the one or more new training data sets is new relative to the first plurality of training data sets;

updating the client training data to include the one or more new training data sets;

generating a plurality of new trained predictive models using the plurality of training functions and a different, third sample of the client training data;

determining, a respective accuracy of each of the plurality of new trained predictive models using a different, fourth sample of the client training data;

generating a respective effectiveness score for each of the plurality of trained predictive models and each of the plurality of new trained predictive models using the determined accuracy of its respective trained predictive model;

receiving, over the network from a client computing system, a first prediction request and first input data;

selecting a first trained predictive model to service the first prediction request from among the plurality of trained predictive models and the plurality of new trained predictive models based on the respective effectiveness scores;

running the first trained predictive model on the first input data to generate a predictive output; and

providing, to the client computing system, the predictive output in response to the first prediction request.

15. The computer-readable storage device of claim 14, wherein the plurality of trained predictive models includes one or more trained updateable predictive models, and wherein the operations further comprise:

generating a retrained updateable predictive model using a previously-trained updateable predictive model, the training function that was used to generate the previously-trained updateable predictive model, and a different, fifth, sample of the client training data;

determining an accuracy of the retrained updateable predictive model using a different, sixth sample of data of the client training data;

generating an effectiveness score for the retrained updateable predictive model using the determined accuracy of the retrained updateable predictive model;

receiving a second prediction request and second input data;

selecting a second trained predictive model to service the second prediction request from among the retrained updateable predictive model and the plurality of trained predictive models based on the respective effectiveness scores; and

running the second trained predictive model on the second input data in response to the second prediction request.

16. The computer-readable storage device of claim 15, wherein the operations further comprise:

determining, before generating the retrained updateable predictive model, that when at least one of the following conditions is true;

(i) an amount of training data in a training data queue is greater than or equal to a threshold amount;

(ii) a predetermined amount of time is reached or exceeded;

or (iii) a request to update the previously-trained updateable predictive model is received.

17. The computer-readable storage device of claim 14, wherein the operations further comprise:

determining, before generating the plurality of new trained predictive models, that at least one of;

(i) a predetermined amount of time is reached or exceeded;

or (ii) a request to generate new trained predictive models is received.

18. The computer-readable storage device of claim 14, wherein the third sample of client training data includes one or more new training data sets and one or more training data sets from the first plurality of training data sets.

19. The computer-readable storage device of claim 14, wherein the third sample of client training data does not include any of the training data sets included in the first plurality of training data sets.

22. The computer-readable storage device of claim 14, wherein the operations further comprise:

determining a respective resource usage for running each of the plurality of trained predictive models; and

generating a respective effectiveness score for each of the plurality of trained predictive models using its respective determined resource usage.