Dynamic predictive modeling platform

• US 8,595,154 B2
• Filed: 01/26/2011
• Issued: 11/26/2013
• Est. Priority Date: 01/26/2011
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Breckenridge, Jordan M., Green, Travis, Kaplow, Robert, Lin, Wei-Hao, Mann, Gideon S.
• Primary Examiner(s): Wong, Lut
• Assistant Examiner(s): FINK, THOMAS ANDREW
• Application Number:
  US13/014,252
• Publication Number:
  US 20120191631A1
• Time in Patent Office: 1,035 Days
• Field of Search: 706/12
• US Class Current: 706/12
• CPC Class Codes: G06N 20/00 Machine learning

Abstract

Methods, systems, and apparatus, including computer programs encoded on one or more computer storage devices, for training and retraining predictive models. A series of training data sets are received and added to a training data queue. In response to a first condition being satisfied, multiple retrained predictive models are generated using the training data queue, multiple updateable trained predictive models obtained from a repository of trained predictive models, and multiple training functions. In response to a second condition being satisfied, multiple new trained predictive models are generated using the training data queue, at least some training data stored in a training data repository and training functions. The new trained predictive models include static trained predictive models and updateable trained predictive models. The repository of trained predictive models is updated with at least some of the retrained predictive models and new trained predictive models.

Claims

1. A system comprising:

one or more computers; and

one or more storage devices coupled to the one or more computers and storing;

a repository of training functions,

a repository of trained predictive models comprising static trained predictive models and updateable trained predictive models,

a training data queue,

a training data repository, and

instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising;

receiving a series of training data sets;

adding the training data sets to the training data queue;

in response to a first condition being satisfied,

generating a plurality of retrained predictive models using the training data queue, a plurality of updateable trained predictive models obtained from the repository of trained predictive models, and a plurality of training functions obtained from the repository of training functions, wherein the first condition is satisfied when a ratio of a size of the training data queue to a size of the training data repository exceeds a predetermined threshold; and

storing one or more of the plurality of generated retrained predictive models in the repository of trained predictive models; and

in response to a second condition being satisfied,

generating a plurality of new trained predictive models using the training data queue, at least some of the training data stored in the training data repository, and a plurality of training functions obtained from the repository of training functions, wherein the plurality of new trained predictive models comprise new static trained predictive models and new updateable trained predictive models; and

storing at least some of the plurality of new trained predictive models in the repository of trained predictive models.

2. The system of claim 1, wherein the series of training data sets are received incrementally.

3. The system of claim 1, wherein the series of training data sets are received together in a batch.

4. The system of claim 1, wherein the second condition is satisfied in response to receiving a command to generate new static models and update the updateable models included in the repository of trained predictive models.

5. The system of claim 1, wherein the second condition is satisfied after a predetermined time period has expired.

6. The system of claim 1, wherein the second condition is satisfied when a size of the training data queue is greater than or equal to a threshold size.

7. The system of claim 1, further comprising:

a user interface configured to receive user input specifying a data retention policy that defines rules for maintaining and deleting training data included in the training data repository.

8. The system of claim 1, where the operations further comprise:

generating updated training data that includes at least some of the training data from the training data queue and at least some of the training data from the training data repository; and

updating the training data repository by storing the updated training data.

9. The system of claim 8, wherein generating updated training data comprises implementing a data retention policy that defines rules for maintaining and deleting training data included in at least one of the training data queue or the training data repository.

10. The system of claim 9, wherein the data retention policy includes a rule for deleting training data from the training data repository when the training data repository size reaches a predetermined size limit.

11. The system of claim 1, wherein, in response to the first condition being satisfied, the operations further comprise:

for each of the plurality of retrained predictive models;

comparing an effectiveness score of the retrained predictive model to an effectiveness score of the updateable trained predictive model from the repository of trained predictive models that was used to generate the retrained predictive model; and

based on the comparison, selecting a first of the two predictive models to store in the repository of trained predictive models and not storing a second of the two predictive models in the repository of trained predictive models;

wherein the effectiveness scores are each scores that represents an estimation of the effectiveness of the respective trained predictive model.

19. The system of claim 1, wherein, in response to the second condition being satisfied, the operations further comprise:

discarding all of the static trained predictive models in the repository of trained predictive models, then storing all of the new static trained predictive models in the repository of trained predictive models.

20. The system of claim 1, wherein, in response to the second condition being satisfied, the operations further comprise:

for each of the new updateable trained predictive models;

comparing an effectiveness score of the new updateable trained predictive model to an effectiveness score of the updateable trained predictive model from the repository of trained predictive models that was used to generate the new updateable trained predictive model;

based on the comparison, selecting a first of the two updateable trained predictive models to store in the repository of trained predictive models and not storing a second of the two updateable trained predictive models in the repository of trained predictive models.

21. The system of claim 1, wherein, in response to the second condition being satisfied, the operation further comprise:

discarding all of the trained predictive models in the repository of trained predictive models prior to storing the plurality of new trained predictive models in the repository of trained predictive models.

12. A computer-implemented method comprising:

receiving new training data;

adding the new training data to a training data queue;

determining whether a size of the training data queue is greater than a threshold;

when the size of the training data queue is greater than the threshold, retrieving a stored plurality of trained predictive models and a stored training data set, wherein each of the trained predictive models were generated using the training data set and a plurality of training functions, and wherein each of the trained predictive models is associated with a score that represents an estimation of the effectiveness of the predictive model;

generating a plurality of retrained predictive models using the training data queue, the retrieved plurality of trained predictive models and the plurality of training functions;

generating a respective new score for each of the generated retrained predictive models; and

adding at least some of the training data queue to the stored training data set,

wherein the threshold is a predetermined ratio of the training data queue size to a size of the stored training data set.

13. A computer-implemented method comprising:

receiving a series of training data sets;

adding the training data sets to a training data queue;

in response to a first condition being satisfied,

generating a plurality of retrained predictive models using the training data queue, a plurality of updateable trained predictive models obtained from a repository of trained predictive models, and a plurality of training functions obtained from a repository of training functions, wherein the first condition is satisfied when a ratio of a size of the training data queue to a size of the training data repository exceeds a predetermined threshold; and

storing one or more of the plurality of generated retrained predictive models in the repository of trained predictive models; and

in response to a second condition being satisfied,

generating a plurality of new trained predictive models using the training data queue, at least some of training data stored in a training data repository, and a plurality of training functions obtained from the repository of training functions, wherein the plurality of new trained predictive models comprise new static trained predictive models and new updateable trained predictive models; and

storing at least some of the plurality of new trained predictive models in the repository of trained predictive models.

14. The method of claim 13, wherein the second condition is satisfied when a predetermined period of time has expired.

15. The method of claim 13, further comprising:

generating updated training data that includes at least some of the training data from the training data queue and at least some of the training data from the training data repository; and

updating the training data repository by storing the updated training data.

22. The method of claim 13, wherein, in response to the second condition being satisfied, the method further comprises:

discarding all of the static trained predictive models in the repository of trained predictive models, then storing all of the new static trained predictive models in the repository of trained predictive models.

23. The method of claim 13, wherein, in response to the second condition being satisfied, the method further comprises:

for each of the new updateable trained predictive models;

comparing an effectiveness score of the new updateable trained predictive model to an effectiveness score of the updateable trained predictive model from the repository of trained predictive models that was used to generate the new updateable trained predictive model;

based on the comparison, selecting a first of the two updateable trained predictive models to store in the repository of trained predictive models and not storing a second of the two updateable trained predictive models in the repository of trained predictive models.

24. The method of claim 13, wherein, in response to the second condition being satisfied, the method further comprises:

discarding all of the trained predictive models in the repository of trained predictive models prior to storing the plurality of new trained predictive models in the repository of trained predictive models.

16. A non-transitory computer-readable storage device encoded with a computer program product, the computer program product comprising instructions that when executed on one or more computers cause the one or more computers to perform operations comprising:

receiving a series of training data sets;

adding the training data sets to a training data queue;

in response to a first condition being satisfied,

generating a plurality of retrained predictive models using the training data queue, a plurality of updateable trained predictive models obtained from a repository of trained predictive models, and a plurality of training functions obtained from a repository of training functions, wherein the first condition is satisfied when a ratio of a size of the training data queue to a size of the training data repository exceeds predetermined threshold; and

storing one or more of the plurality of generated retrained predictive models;

in response to a second condition being satisfied,

generating a plurality of new trained predictive models using the training data queue, at least some of training data stored in a training data repository, and a plurality of training functions obtained from the repository of training functions, wherein the plurality of new trained predictive models comprise new static trained predictive models and new updateable trained predictive models; and

storing at least some of the plurality of new trained predictive models in the repository of trained predictive models.

17. The computer-readable storage device of claim 16, wherein the second condition is satisfied when a predetermined period of time has expired.

18. The computer-readable storage device of claim 16, the operations further comprising:

generating updated training data that includes at least some of the training data from the training data queue and at least some of the training data from the training data repository; and

updating the training data repository by storing the updated training data.

25. The computer-readable storage device of claim 16, wherein, in response to the second condition being satisfied, the operations further comprise:

discarding all of the static trained predictive models in the repository of trained predictive models, then storing all of the new static trained predictive models in the repository of trained predictive models.

26. The computer-readable storage device of claim 16, wherein, in response to the second condition being satisfied, the operations further comprise:

for each of the new updateable trained predictive models;

comparing an effectiveness score of the new updateable trained predictive model to an effectiveness score of the updateable trained predictive model from the repository of trained predictive models that was used to generate the new updateable trained predictive model;

based on the comparison, selecting a first of the two updateable trained predictive models to store in the repository of trained predictive models and not storing a second of the two updateable trained predictive models in the repository of trained predictive models.

27. The computer-readable storage device of claim 16, wherein, in response to the second condition being satisfied, the operations further comprise:

discarding all of the trained predictive models in the repository of trained predictive models prior to storing the plurality of new trained predictive models in the repository of trained predictive models.

28. A system comprising:

one or more computers; and

one or more storage devices coupled to the one or more computers and storing;

training functions,

trained predictive models, wherein each trained predictive model is associated with a respective score that represents an estimation of the effectiveness of the trained predictive model,

a training data queue,

a training data set, and

instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising;

receiving new training data;

adding the new training data to the training data queue;

determining whether a size of the training data queue is greater than a threshold;

when the size of the training data queue is greater than the threshold, retrieving the trained predictive models and the training data set, wherein each of the trained predictive models was generated using the training data set and the training functions;

generating retrained predictive models using the training data queue, the trained predictive models, and the training functions;

generating a respective new score for each of the generated retrained predictive models; and

adding at least some of the training data queue to the training data set,

wherein the threshold is a predetermined ratio of a size of the training data queue to a size of the training data set.

29. A non-transitory computer-readable storage device encoded with a computer program product, the computer program product comprising instructions that when executed on one or more computers cause the one or more computers to perform operations comprising:

receiving new training data;

adding the new training data to a training data queue;

determining whether a size of the training data queue is greater than a threshold;

when the size of the training data queue is greater than the threshold, retrieving a stored plurality of trained predictive models and a stored training data set, wherein each of the trained predictive models were generated using the training data set and a plurality of training functions, and wherein each of the trained predictive models is associated with a score that represents an estimation of the effectiveness of the predictive model;

generating a plurality of retrained predictive models using the training data queue, the retrieved plurality of trained predictive models and the plurality of training functions;

generating a respective new score for each of the generated retrained predictive models; and

adding at least some of the training data queue to the stored training data set,

wherein the threshold is a predetermined ratio of the training data queue size to a size of the stored training data set.