Predictive analytic modeling platform

• US 9,189,747 B2
• Filed: 07/20/2012
• Issued: 11/17/2015
• Est. Priority Date: 05/14/2010
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Mann, Gideon S., Breckenridge, Jordan M., Lin, Wei-Hao
• Primary Examiner(s): Gaffin, Jeffrey A
• Assistant Examiner(s): BROWN JR, NATHAN H
• Application Number:
  US13/554,942
• Publication Number:
  US 20150170049A1
• Time in Patent Office: 1,215 Days
• Field of Search: 706/12, 706/21
• US Class Current: 1/1
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 5/04 Inference or reasoning models

Abstract

Methods, systems, and apparatus, including computer programs encoded on one or more computer storage devices, for training a predictive model. In one aspect, a method includes receiving over a network predictive modeling training data from a client computing system. The training data and multiple training functions obtained from a repository of training functions are used to train multiple predictive models. A score is generated for each of the trained predictive models, where each score represents an estimation of the effectiveness of the respective trained predictive model. A first trained predictive model is selected from among the trained predictive models based on the generated scores. Access to the first trained predictive model is provided over the network.

Claims

1. A computer-implemented method comprising:

training each of a plurality of predictive models using training data, wherein the predictive models include two or more predictive models of a same type that are trained with different combinations of features of the training data;

generating, for each of the plurality of trained predictive models, a respective score that represents an estimation of an effectiveness of the respective trained predictive model;

receiving a request for a prediction that includes input data from a client system;

in response to receiving the request for the prediction, selecting a first subset of the plurality of trained predictive models based on the respective scores of the trained predictive models in the first subset, wherein the plurality of trained predictive models includes the first subset and a second subset, each subset comprises at least one trained predictive model, the first subset and the second subset are disjoint sets, and the predictive models in the first subset have higher respective scores than predictive models that were not selected;

obtaining a respective predictive output from only each of the selected predictive models in the first subset based on the request and using the input data;

combining the predictive outputs to generate a result; and

providing the result to the client system.

2. The method of claim 1 wherein training each of the plurality of predictive models using the training data further comprising:

partitioning the training data into k partitions where k is an integer greater than 1;

performing k-fold cross-validation of the predictive model using the k partitions; and

using a cross-validation score for the predictive model as the respective score for the predictive model.

3. The method of claim 1 wherein the respective score is based on, at least, an estimate of resource usage costs for the trained model.

4. The method of claim 1 wherein training each of the plurality of predictive models using the training data comprises training the predictive model using all of the training data.

5. The method of claim 1, further comprising receiving the training data from a client computing system.

6. The method of claim 1, further comprising allocating a fee obtained for use of the predictive models in the first subset to one or more respective parties associated with the predictive models in the first subset.

7. The method of claim 1 wherein combining the predictive outputs to generate the result further comprises averaging the predictive outputs to generate the result.

8. The method of claim 1 wherein the plurality of trained predictive models includes a Naï

ve Bayes model, a Perceptron model, a Support Vector Machine model, a linear regression model, a k-nearest neighbor model, or a logistic regression model.

9. The method of claim 1, wherein:

receiving the request for the prediction that includes the input data comprises receiving the request for a predictive output of a certain type; and

selecting the first subset of the plurality of trained predictive models based on the respective scores of the trained predictive models in the first subset comprises selecting the first subset of the plurality of trained predictive models using the certain type identified in the request.

10. A system comprising:

one or more computers; and

one or more data storage devices having instructions stored thereon that, when executed by the computers, cause the computers to perform operations comprising;

training each of a plurality of predictive models using training data, wherein the predictive models include two or more predictive models of a same type that are trained with different combinations of features of the training data;

generating, for each of the plurality of trained predictive models, a respective score that represents an estimation of an effectiveness of the respective trained predictive model;

receiving a request for a prediction that includes input data from a client system;

in response to receiving the request for the prediction, selecting a first subset of the plurality of trained predictive models based on the respective scores of the trained predictive models in the first subset, wherein the plurality of trained predictive models includes the first subset and a second subset, each subset comprises at least one trained predictive model, the first subset and the second subset are disjoint sets, and the predictive models in the first subset have higher respective scores than predictive models that were not selected;

obtaining a respective predictive output from only each of the selected predictive models in the first subset based on the request and using the input data;

combining the predictive outputs to generate a result; and

providing the result to the client system.

11. The system of claim 10 wherein training each of the plurality of predictive models using the training data further comprising:

partitioning the training data into k partitions where k is an integer greater than 1;

performing k-fold cross-validation of the predictive model using the k partitions; and

using a cross-validation score for the predictive model as the respective score for the predictive model.

12. The system of claim 10 wherein the respective score is based on, at least, an estimate of resource usage costs for the trained model.

13. The system of claim 10 wherein training each of the plurality of predictive models using the training data comprises training the predictive model using all of the training data.

14. The system of claim 10, further comprising receiving the training data from a client computing system.

15. The system of claim 10, further comprising allocating a fee obtained for use of the predictive models in the first subset to one or more respective parties associated with the predictive models in the first subset.

16. The system of claim 10 wherein combining the predictive outputs to generate the result further comprises averaging the predictive outputs to generate the result.

17. The system of claim 10 wherein the plurality of trained predictive models includes a Naï

ve Bayes model, a Perceptron model, a Support Vector Machine model, a linear regression model, a k-nearest neighbor model, or a logistic regression model.

18. The system of claim 10, wherein:

receiving the request for the prediction that includes the input data comprises receiving the request for a predictive output of a certain type; and

selecting the first subset of the plurality of trained predictive models based on the respective scores of the trained predictive models in the first subset comprises selecting the first subset of the plurality of trained predictive models using the certain type identified in the request.

19. A non-transitory computer-readable storage device encoded with instructions which, when executed by one or more computers, cause the one or more computers to perform operations comprising:

training each of a plurality of predictive models using training data, wherein the predictive models include two or more predictive models of a same type that are trained with different combinations of features of the training data;

generating, for each of the plurality of trained predictive models, a respective score that represents an estimation of an effectiveness of the respective trained predictive model;

receiving a request for a prediction that includes input data from a client system;

in response to receiving the request for the prediction, selecting a first subset of the plurality of trained predictive models based on the respective scores of the trained predictive models in the first subset, wherein the plurality of trained predictive models includes the first subset and a second subset, each subset comprises at least one trained predictive model, the first subset and the second subset are disjoint sets, and the predictive models in the first subset have higher respective scores than predictive models that were not selected;

obtaining a respective predictive output from only each of the selected predictive models in the first subset based on the request and using the input data;

combining the predictive outputs to generate a result; and

providing the result to the client system.

20. The storage device of claim 19 wherein training each of the plurality of predictive models using the training data further comprising:

partitioning the training data into k partitions where k is an integer greater than 1;

performing k-fold cross-validation of the predictive model using the k partitions; and

using a cross-validation score for the predictive model as the respective score for the predictive model.

21. The storage device of claim 19 wherein the respective score is based on, at least, an estimate of resource usage costs for the trained model.

22. The storage device of claim 19 wherein training each of the plurality of predictive models using the training data comprises training the predictive model using all of the training data.

23. The storage device of claim 19, further comprising receiving the training data from a client computing storage device.

24. The storage device of claim 19, further comprising allocating a fee obtained for use of the predictive models in the first subset to one or more respective parties associated with the predictive models in the first subset.

25. The storage device of claim 19 wherein combining the predictive outputs to generate the result further comprises averaging the predictive outputs to generate the result.

26. The storage device of claim 19 wherein the plurality of trained predictive models includes a Naï

ve Bayes model, a Perceptron model, a Support Vector Machine model, a linear regression model, a k-nearest neighbor model, or a logistic regression model.

27. The storage device of claim 19, wherein:

receiving the request for the prediction that includes the input data comprises receiving the request for a predictive output of a certain type; and

selecting the first subset of the plurality of trained predictive models based on the respective scores of the trained predictive models in the first subset comprises selecting the first subset of the plurality of trained predictive models using the certain type identified in the request.