Predictive analytic modeling platform

• US 8,909,568 B1
• Filed: 03/04/2014
• Issued: 12/09/2014
• Est. Priority Date: 05/14/2010
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Mann, Gideon S., Breckenridge, Jordan M., Lin, Wei-Hao
• Primary Examiner(s): Holmes, Michael B
• Application Number:
  US14/196,555
• Time in Patent Office: 280 Days
• Field of Search: None
• US Class Current: 706/12
• CPC Class Codes: G06N 20/00 Machine learning

Abstract

Methods, systems, and apparatus, including computer programs encoded on one or more computer storage devices, for training a predictive model. In one aspect, a method includes receiving over a network predictive modeling training data from a client computing system. The training data and multiple training functions obtained from a repository of training functions are used to train multiple predictive models. A score is generated for each of the trained predictive models, where each score represents an estimation of the effectiveness of the respective trained predictive model. A first trained predictive model is selected from among the trained predictive models based on the generated scores. Access to the first trained predictive model is provided to the client computing system.

Claims

1. A computer-implemented method comprising:

receiving over a network predictive modeling training data from a client computing system;

partitioning the training data into a plurality of subsamples;

using the plurality of subsamples and a plurality of training functions obtained from a repository of training functions to train a plurality of predictive models using cross-validation;

generating a cross-validation score for each of the plurality of trained predictive models, where each cross-validation score indicates the accuracy of the respective trained predictive model;

selecting a first trained predictive model from among the plurality of trained predictive models using the generated cross-validation scores; and

providing access to the first trained predictive model over the network.

2. The method of claim 1, wherein the plurality of subsamples comprises k subsamples and providing access to the first trained predictive model comprises providing access to the first trained predictive model trained with all k subsamples from the plurality of subsamples.

3. The method of claim 2, wherein using the plurality of subsamples and the plurality of training functions obtained from the repository of training functions to train the plurality of predictive models using cross-validation comprises:

for each iteration of training;

training each of the plurality of predictive models using a different subset of subsamples from the plurality of subsamples comprising k−

1 subsamples;

determining, for each of the trained predictive models, a similarity between an output from the respective trained predictive model and an expected output identified in another subsample not included in the respective subset of subsamples; and

generating the cross-validation score for each of the plurality of trained predictive models using the respective similarity.

4. The method of claim 3, comprising:

averaging, for each of the trained predictive models, the cross-validation scores generated for the respective trained predictive model, wherein selecting the first trained predictive model from among the plurality of trained predictive models using the generated cross-validation scores comprises selecting the first trained predictive model from among the plurality of trained predictive models using the averages of the generated cross-validation scores.

5. The method of claim 1, comprising:

receiving input data, data identifying the first trained predictive model, and a request for a predictive output; and

generating the predictive output using the first predictive model and the input data.

6. The method of claim 1, wherein the plurality of training functions includes two or more training functions for training predictive models with a same type of predictive output and a same type of input data, where each predictive model is trained with a different training function.

7. The method of claim 1, wherein the plurality of training functions includes two or more training functions for training predictive models with a same type of predictive output and a same type of input data, where each predictive model is trained with a different hyper-parameter configuration.

8. The method of claim 1, wherein providing access to the first trained predictive model comprises providing a universal resource locator (URL) that identifies the first trained predictive model as an addressable resource.

9. The method of claim 1, wherein the cross-validation comprises k-fold cross-validation.

10. A system, comprising:

a data processing apparatus; and

a non-transitory computer readable storage medium in data communication with the data processing apparatus and storing instructions executable by the data processing apparatus and upon such execution cause the data processing to perform operations comprising;

receiving over a network predictive modeling training data from a client computing system;

partitioning the training data into a plurality of subsamples;

using the plurality of subsamples and a plurality of training functions obtained from a repository of training functions to train a plurality of predictive models using cross-validation;

generating a cross-validation score for each of the plurality of trained predictive models, where each cross-validation score indicates the accuracy of the respective trained predictive model;

selecting a first trained predictive model from among the plurality of trained predictive models using the generated cross-validation scores; and

providing access to the first trained predictive model over the network.

11. The system of claim 10, wherein the plurality of subsamples comprises k subsamples and providing access to the first trained predictive model comprises providing access to the first trained predictive model trained with all k subsamples from the plurality of subsamples.

12. The system of claim 11, wherein using the plurality of subsamples and the plurality of training functions obtained from the repository of training functions to train the plurality of predictive models using cross-validation comprises:

for each iteration of training;

training each of the plurality of predictive models using a different subset of subsamples from the plurality of subsamples comprising k−

1 subsamples;

determining, for each of the trained predictive models, a similarity between an output from the respective trained predictive model and an expected output identified in another subsample not included in the respective subset of subsamples; and

generating the cross-validation score for each of the plurality of trained predictive models using the respective similarity.

13. The system of claim 12, the operations comprising:

averaging, for each of the trained predictive models, the cross-validation scores generated for the respective trained predictive model, wherein selecting the first trained predictive model from among the plurality of trained predictive models using the generated cross-validation scores comprises selecting the first trained predictive model from among the plurality of trained predictive models using the averages of the generated cross-validation scores.

14. The system of claim 10, the operations comprising:

receiving input data, data identifying the first trained predictive model, and a request for a predictive output; and

generating the predictive output using the first predictive model and the input data.

15. The system of claim 10, wherein the plurality of training functions includes two or more training functions for training predictive models with a same type of predictive output and a same type of input data, where each predictive model is trained with a different training function.

16. The system of claim 10, wherein the plurality of training functions includes two or more training functions for training predictive models with a same type of predictive output and a same type of input data, where each predictive model is trained with a different hyper-parameter configuration.

17. The system of claim 10, wherein providing access to the first trained predictive model comprises providing a universal resource locator (URL) that identifies the first trained predictive model as an addressable resource.

18. The system of claim 10, wherein the cross-validation comprises k-fold cross-validation.

19. A non-transitory computer readable storage medium storing instructions executable by a data processing apparatus and upon such execution cause the data processing to perform operations comprising:

receiving over a network predictive modeling training data from a client computing system;

partitioning the training data into a plurality of subsamples;

using the plurality of subsamples and a plurality of training functions obtained from a repository of training functions to train a plurality of predictive models using cross-validation;

generating a cross-validation score for each of the plurality of trained predictive models, where each cross-validation score indicates the accuracy of the respective trained predictive model;

selecting a first trained predictive model from among the plurality of trained predictive models using the generated cross-validation scores; and

providing access to the first trained predictive model over the network.

20. The computer readable storage medium of claim 19, wherein the plurality of training functions includes two or more training functions for training predictive models with a same type of predictive output and a same type of input data, where each predictive model is trained with a different hyper-parameter configuration.