Combining predictive models in predictive analytical modeling

• US 8,370,280 B1
• Filed: 10/03/2011
• Issued: 02/05/2013
• Est. Priority Date: 07/14/2011
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Lin, Wei-Hao, Green, Travis H., Kaplow, Robert, Fu, Gang, Mann, Gideon S.
• Primary Examiner(s): Chaki, Kakali
• Assistant Examiner(s): Gonzales, Vincent
• Application Number:
  US13/252,063
• Time in Patent Office: 491 Days
• Field of Search: 706/45, 706/12
• US Class Current: 706/12
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 5/04 Inference or reasoning models

Abstract

A method can include the actions of: receiving a feature vector, the feature vector including one or more elements; identifying an element type for each of the one or more elements; selecting, from a set of predictive models, a subset of one or more predictive models based on the element types and one or more performance indicators associated with each predictive model in the set of predictive models; processing the feature vector using the subset of predictive models, each predictive model of the subset of predictive models generating an output based on the feature vector to provide a plurality of outputs; and generating a final output based on the plurality of outputs. Other embodiments may include corresponding systems, apparatus, and computer program products for executing the method.

Claims

1. A computer-implemented method, comprising:

storing, at a server system, a set of previously trained predicative models;

storing, at the server system, a respective performance indicator associated with each predictive model in the set of predictive models, where each of the respective performance indicators comprises a quantifiable metric determined based on prior usage data corresponding to the associated predictive model;

receiving, at the server system, a first feature vector from a first remote computing device, the first feature vector comprising one or more elements;

identifying, using the server system, an element type for each of the one or more elements of the first feature vector;

selecting, using the server system, a first subset of predictive models from the set of predictive models, where the selection is based on the identified element types of the first feature vector and the stored performance indicators associated with the predictive models of the set;

processing the first feature vector using the first subset of predictive models, each predictive model of the first subset of predictive models generating a respective predictive output based on the first feature vector to provide a first plurality of predictive outputs;

generating a first combined predictive output based on the first plurality of predictive outputs;

in response to generating the first combined predictive output, evaluating a performance of at least one predictive model of the subset; and

updating the performance indicator associated with the at least one predictive model based on the evaluated performance.

2. The method of claim 1, wherein generating a first combined predictive output comprises:

defining a combining technique based on the first subset of predictive models; and

combining the first plurality of predictive outputs according to the combing technique.

3. The method of claim 2, wherein the combining technique comprises a fixed output combining rule.

4. The method of claim 2, wherein the combining technique comprises a trained combining rule.

5. The method of claim 2, wherein the combining technique is defined as a directed acyclic graph.

6. The method of claim 1, wherein each predictive output of the first plurality of predictive outputs is equally weighted when generating the first combined predictive output.

7. The method of claim 1, wherein the first combined predictive output is generated using a decision maker, the decision maker receiving the first plurality of predictive outputs.

8. The method of claim 1, wherein the first plurality of outputs are combined to define a combined feature vector, the combined feature vector being processed by a combining predictive model to generate the combined predictive output.

9. The method of claim 1, wherein selecting a first subset of one or more predictive models comprises comparing respective performance indicators associated with each predictive model of the set of predictive models, the respective performance indicators being selected for comparison based on the element types of the first feature vector.

10. The method of claim 9, wherein comparing respective performance indicators comprises comparing accuracy metrics.

11. The method of claim 1, wherein processing the first feature vector comprises submitting at least a portion of the first feature vector to each predictive model of the first subset of predictive models in parallel.

12. The method of claim 1, wherein processing the first feature vector comprises submitting the entire first feature vector to each predictive model of the first subset of predictive models.

13. The method of claim 1, wherein processing the first feature vector comprises submitting elements of a first type to a first predictive model of the first subset of predictive models and submitting elements of a second type to a second predictive model of the first subset of predictive models.

14. The method of claim 1, further comprising:

receiving, at the server system, a second feature vector from a second remote computing device, the second feature vector comprising one or more elements;

identifying, using the server system, an element type for each of the one or more elements of the second feature vector;

selecting, using the server system, a second subset of predictive models from the set of predictive models, where the selection is based on the identified element types of the second feature vector and the stored performance indicators associated with the predictive models of the set, wherein the second subset of predictive models is different than the first subset of predictive models, and wherein the stored performance indicators comprise the updated performance indicator;

processing the second feature vector using the second subset of predictive models, each predictive model of the second subset of predictive models generating a respective predictive output based on the second feature vector to provide a second plurality of predictive outputs; and

generating a second combined predictive output based on the second plurality of predictive outputs.

15. A computer storage medium encoded with a computer program, the program comprising instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:

storing a set of previously trained predicative models;

storing a respective performance indicator associated with each predictive model in the set of predictive models, where each of the respective performance indicators comprises a quantifiable metric determined based on prior usage data corresponding to the associated predictive model;

receiving a first feature vector from a first remote computing device, the first feature vector comprising one or more elements;

identifying an element type for each of the one or more elements of the first feature vector;

selecting a first subset of one or more predictive models from the stored set of predictive models, where the selection is based on the identified element types of the first feature vector and the stored performance indicators associated with the predictive models of the set;

processing the first feature vector using the first subset of predictive models, each predictive model of the first subset of predictive models generating a respective predictive output based on the first feature vector to provide a first plurality of predictive outputs;

generating a first combined predictive output based on the first plurality of predictive outputs;

in response to generating the first combined predictive output, evaluating a performance of at least one predictive model of the subset; and

updating the performance indicator associated with the at least one predictive model based on the evaluated performance.

16. The computer storage medium of claim 15, wherein the program instructions cause the one or more computers to perform operations further comprising:

receiving, a second feature vector from a second remote computing device, the second feature vector comprising one or more elements;

identifying an element type for each of the one or more elements of the second feature vector;

selecting a second subset of predictive models from the stored set of predictive models, where the selection is based on the identified element types of the second feature vector and the stored performance indicators associated with the predictive model of the set, wherein the second subset of predictive models is different than the first subset of predictive models, and wherein the stored performance indicators comprise the updated performance indicator;

processing the second feature vector using the second subset of predictive models, each predictive model of the second subset of predictive models generating a respective predictive output based on the second feature vector to provide a second plurality of predictive outputs; and

generating a second combined predictive output based on the second plurality of predictive outputs.

17. A system, comprising:

one or more computing devices; and

one or more computer-readable media coupled to the one or more computing devices and having instructions stored thereon which, when executed by the one or more computing devices, cause the one or more computing devices to perform operations comprising;

storing a set of previously trained predicative models;

storing a respective performance indicator associated with each predictive model in the set of predictive models, where each of the respective performance indicators comprises a quantifiable metric determined based on prior usage data corresponding to the associated predictive model;

receiving a first feature vector from a first remote computing device, the first feature vector comprising one or more elements;

identifying an element type for each of the one or more elements of the first feature vector;

selecting a first subset of one or more predictive models from the stored set of predictive models, where the selection is based on the identified element types of the first feature vector and the stored performance indicators associated with the predictive models of the set;

processing the first feature vector using the first subset of predictive models, each predictive model of the first subset of predictive models generating a respective predictive output based on the first feature vector to provide a first plurality of predictive outputs;

generating a first combined predictive output based on the first plurality of predictive outputs;

in response to generating the first combined predictive output, evaluating a performance of at least one predictive model of the subset; and

updating the performance indicator associated with the at least one predictive model based on the evaluated performance.

18. The system of claim 17, wherein the stored instructions cause the one or more computers to perform operations further comprising:

receiving, a second feature vector from a second remote computing device, the second feature vector comprising one or more elements;

identifying an element type for each of the one or more elements of the second feature vector;

selecting a second subset of predictive models from the stored set of predictive models, where the selection is based on the identified element types of the second feature vector and the stored performance indicators associated with the predictive model of the set, wherein the second subset of predictive models is different than the first subset of predictive models, and wherein the stored performance indicators comprise the updated performance indicator;

processing the second feature vector using the second subset of predictive models, each predictive model of the second subset of predictive models generating a respective predictive output based on the second feature vector to provide a second plurality of predictive outputs; and

generating a second combined predictive output based on the second plurality of predictive outputs.