Predictive analytic modeling platform

• US 8,473,431 B1
• Filed: 05/18/2010
• Issued: 06/25/2013
• Est. Priority Date: 05/14/2010
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Mann, Gideon S., Breckenridge, Jordan M., Lin, Wei-Hao
• Primary Examiner(s): Gaffin, Jeffrey A
• Assistant Examiner(s): BROWN JR, NATHAN H
• Application Number:
  US12/782,689
• Time in Patent Office: 1,134 Days
• Field of Search: 706/12
• US Class Current: 706/12
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 5/04 Inference or reasoning models

Abstract

Methods, systems, and apparatus, including computer programs encoded on one or more computer storage devices, for training a predictive model. In one aspect, a method includes receiving over a network predictive modeling training data from a client computing system. The training data and multiple training functions obtained from a repository of training functions are used to train multiple predictive models. A score is generated for each of the trained predictive models, where each score represents an estimation of the effectiveness of the respective trained predictive model. A first trained predictive model is selected from among the trained predictive models based on the generated scores. Access to the first trained predictive model is provided over the network.

Claims

1. A system comprising:

one or more computers;

one or more data storage devices coupled to the one or more computers;

the one or more data storage devices storing a repository of training functions and further storing instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising;

receiving over a network predictive modeling training data from a client computing system;

using the training data and a plurality of training functions obtained from the repository to train a plurality of predictive models, wherein the plurality of training functions includes two or more training functions for training predictive models of a same type, and wherein predictive models of the same type are trained with different hyper-parameter configurations;

generating a score for each of the plurality of trained predictive models, where each score represents an estimation of the effectiveness of the respective trained predictive model;

selecting a first trained predictive model from among the plurality of trained predictive models based on the generated scores; and

providing access to the first trained predictive model over the network.

2. The system of claim 1, wherein:

using the training data to train each of the plurality of predictive models and generating a score for each of the plurality of trained predictive models comprises partitioning the training data into k partitions, performing k-fold cross-validation and generating a cross-validation score for each of the plurality of trained predictive models that indicates the accuracy of the trained predictive model, where k is an integer greater than 1.

3. The system of claim 2, wherein providing access to the first trained predictive model comprises providing access to the first trained predictive model trained with all k partitions of the training data.

4. The system of claim 1, where the operations further comprise:

receiving input data, data identifying the first trained predictive model, and a request for a predictive output; and

generating the predictive output using the first trained predictive model and the input data.

5. The system of claim 1, wherein providing access to the first trained predictive model comprises providing a universal resource locator (URL) to that identifies the first trained predictive model as an addressable resource.

6. The system of claim 1, wherein the operations further comprise:

receiving over the network a training request over an HTTP connection at a universal resource locator (URL) address.

7. The system of claim 1, wherein generating a score for each of the plurality of trained predictive models includes generating an estimate of resource usage costs for each of the plurality of trained predictive models and generating the score based at least in part on the estimates.

8. The system of claim 1, wherein the training data comprises at least a gigabyte of training data.

9. The system of claim 1, wherein the training data comprises at least a terabyte of training data.

10. The system of claim 1, wherein the plurality of trained predictive models includes a Naï

ve Bayes model.

11. The system of claim 1, wherein the plurality of trained predictive models includes a Perceptron model.

12. The system of claim 1, wherein the plurality of trained predictive models includes a Support Vector Machine model.

13. The system of claim 1, wherein the plurality of trained predictive models includes a linear regression model.

14. The system of claim 1, wherein the plurality of trained predictive models includes a logistic regression model.

15. The system of claim 1, wherein the plurality of trained predictive models includes a k-nearest neighbor model.

16. A system comprising:

one or more computers;

one or more data storage devices coupled to the one or more computers,

the one or more data storage devices storing a repository of training functions and further storing instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising;

receiving over a network a plurality of sets of predictive modeling training data from a first plurality of client computing systems;

for each set of predictive modeling training data;

using the training data and a plurality of training functions obtained from the repository to train a plurality of predictive models;

generating a score for each of the plurality of trained predictive models, wherein the score represents an estimation of the effectiveness of the respective trained predictive model; and

selecting a trained predictive model from among the plurality of trained predictive models based on the generated scores;

providing access to the plurality of trained predictive models over the network; and

receiving data from a first client computing system of the first plurality of client computing systems indicating that permission is granted to a second client computing system of a second plurality of client computing systems for the second client computing system to access the selected trained predictive model that was trained using training data received from the first client computing system.

17. A computer-implemented method comprising:

receiving over a network predictive modeling training data from a client computing system;

using the training data and a plurality of training functions obtained from a repository of training functions to train a plurality of predictive models, wherein the plurality of training functions includes two or more training functions for training predictive models of a same type, and wherein predictive models of the same type are trained with different hyper-parameter configurations;

generating a score for each of the plurality of trained predictive models, where each score represents an estimation of the effectiveness of the respective trained predictive model;

selecting a first trained predictive model from among the plurality of trained predictive models based on the generated scores; and

providing access to the first trained predictive model over the network.

18. The method of claim 17, wherein:

using the training data to train each of the plurality of predictive models and generating a score for each of the plurality of trained predictive models comprises partitioning the training data into k partitions, performing k-fold cross-validation and generating a cross-validation score for each of the plurality of trained predictive models that indicates the accuracy of the trained predictive model, where k is an integer greater than 1.

19. The method of claim 18, wherein providing access to the first trained predictive model comprises providing access to the first trained predictive model trained with all k partitions of the training data.

20. The method of claim 17, further comprising:

receiving input data, data identifying the first trained predictive model, and a request for a predictive output; and

generating the predictive output using the first trained predictive model and the input data.

21. The method of claim 17, wherein providing access to the first trained predictive model comprises providing a universal resource locator (URL) that identifies the first trained predictive model as an addressable resource.

22. The method of claim 17, wherein generating a score for each of the plurality of trained predictive models includes generating an estimate of resource usage costs for each of the plurality of trained predictive models and generating the score based at least in part on the estimates.

23. A non-transitory computer-readable storage device encoded with instructions which, when executed by one or more computers, cause the one or more computers to perform operations comprising:

receiving over a network predictive modeling training data from a client computing system;

using the training data and a plurality of training functions obtained from a repository of training functions to train a plurality of predictive models, wherein the plurality of training

functions includes two or more training functions for training predictive models of a same type, and wherein predictive models of the same type are trained with different hyper-parameter configurations;

generating a score for each of the plurality of trained predictive models, where each score represents an estimation of the effectiveness of the respective trained predictive model;

selecting a first trained predictive model from among the plurality of trained predictive models based on the generated scores; and

providing access to the first trained predictive model over the network.

24. The non-transitory computer-readable storage device of claim 23, wherein:

using the training data to train each of the plurality of predictive models and generating a score for each of the plurality of trained predictive models comprises partitioning the training

data into k partitions, performing k-fold cross-validation and generating a cross-validation score for each of the plurality of trained predictive models that indicates the accuracy of the trained predictive model, where k is an integer greater than 1.

25. The non-transitory computer-readable storage device of claim 24, wherein providing access to the first trained predictive model comprises providing access to the first trained predictive model trained with all k partitions of the training data.

26. The non-transitory computer-readable storage device of claim 23, wherein the operations further comprise:

receiving input data, data identifying the first trained predictive model, and a request for a predictive output; and

generating the predictive output using the first trained predictive model and the input data.

27. The non-transitory computer-readable storage device of claim 23, wherein providing access to the first trained predictive model comprises providing a universal resource locator (URL) that identifies the first trained predictive model as an addressable resource.

28. The non-transitory computer-readable storage device of claim 23, wherein generating a score for each of the plurality of trained predictive models includes generating an estimate of resource usage costs for each of the plurality of trained predictive models and generating the score based at least in part on the estimates.

29. A computer-implemented method comprising:

receiving over a network a plurality of sets of predictive modeling training data from a first plurality of client computing systems;

for each set of predictive modeling training data;

using the training data and a plurality of training functions obtained from a repository of training functions to train a plurality of predictive models;

generating a score for each of the plurality of trained predictive models, wherein the score represents an estimation of the effectiveness of the respective trained predictive model; and

selecting a trained predictive model from among the plurality of trained predictive models based on the generated scores;

providing access to the plurality of trained predictive models over the network; and

receiving data from a first client computing system of the first plurality of client computing systems indicating that permission is granted to a second client computing system of a second plurality of client computing systems for the second client computing system to access the selected trained predictive model that was trained using training data received from the first client computing system.

30. A non-transitory computer-readable storage device encoded with instructions which, when executed by one or more computers, cause the one or more computers to perform operations comprising:

receiving over a network a plurality of sets of predictive modeling training data from a first plurality of client computing systems;

for each set of predictive modeling training data;

using the training data and a plurality of training functions obtained from a repository of training functions to train a plurality of predictive models;

generating a score for each of the plurality of trained predictive models, wherein the score represents an estimation of the effectiveness of the respective trained predictive model; and

selecting a trained predictive model from among the plurality of trained predictive models based on the generated scores;

providing access to the plurality of trained predictive models over the network; and

receiving data from a first client computing system of the first plurality of client computing systems indicating that permission is granted to a second client computing system of a second plurality of client computing systems for the second client computing system to access the selected trained predictive model that was trained using training data received from the first client computing system.