Normalization of predictive model scores

• US 10,504,024 B2
• Filed: 06/28/2016
• Issued: 12/10/2019
• Est. Priority Date: 09/29/2011
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Lin, Wei-Hao, Green, Travis H. K., Kaplow, Robert, Fu, Gang, Mann, Gideon S.
• Primary Examiner(s): Wong, Lut
• Application Number:
  US15/194,764
• Publication Number:
  US 20160307099A1
• Time in Patent Office: 1,260 Days
• Field of Search: 706 45
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 5/02 Knowledge representation; Symbolic representation
• CPC Class Codes: G06N 5/022 Knowledge engineering; Knowledge acquisition

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for score normalization. One of the methods includes receiving initial training data, the initial training data comprising initial training records, each initial training record identifying input data as input and a category as output. The method includes generating a first trained predictive model using the initial training data and a training function. The method includes generating intermediate training records by inputting input data of the initial training records to a second trained predictive model, the second trained predictive model generated using the training function, each intermediate training record having a score. The method also includes generating a score normalization model using a score normalization training function and the intermediate training records.

Claims

1. A computer-implemented method comprising:

receiving a predictive request;

identifying a predictive model based on the predictive request;

generating a result using the predictive model and the predictive request; and

generating a normalized result using a score normalization model and the result, the score normalization model being generated using intermediate training records output by the predictive model and additional intermediate training records output by one or more additional predictive models, wherein;

the predictive model is trained using a first subset of initial training records and the intermediate training records are generated by the predictive model using a second subset of the initial training records;

each additional predictive model is trained using a respective third subset of the initial training records and the additional training records output by the additional predictive model are generated by the additional predictive model using a respective fourth subset of the initial training records; and

the first subset of the initial training records is different from each third subset of the initial training records and the second subset of the initial training records is different from each fourth subset of the initial training records.

2. The method of claim 1, further comprising updating the predictive model in response to receiving additional training data, including training the predictive model using the additional training data and a different training function than a training function previously used to train the predictive model.

3. The method of claim 1, wherein the predictive model is trained using a first training function and at least one of the one or more additional predictive models is trained using a second training function different from the first training function.

4. The method of claim 1, wherein identifying a predictive model based on the predictive request comprise identifying the predictive model based on a score for the predictive model and a respective score for each of the one or more additional predictive models, the score for each predictive model being based on a measure of effectiveness of the predictive model.

5. The method of claim 1, wherein the predictive request includes input data that is input to the predictive model to generate the result.

6. A system, comprising:

a data processing apparatus; and

a memory storage apparatus in data communication with the data processing apparatus, the memory storage apparatus storing instructions executable by the data processing apparatus and that upon such execution cause the data processing apparatus to perform operations comprising;

receiving a predictive request;

identifying a predictive model based on the predictive request;

generating a result using the predictive model and the predictive request; and

generating a normalized result using a score normalization model and the result, the score normalization model being generated using intermediate training records output by the predictive model and intermediate training records output by one or more additional predictive models, wherein;

the predictive model is trained using a first subset of initial training records and the intermediate training records are generated by the predictive model using a second subset of the initial training records;

each additional predictive model is trained using a respective third subset of the initial training records and the additional training records output by the additional predictive model are generated by the additional predictive model using a respective fourth subset of the initial training records; and

the first subset of the initial training records is different from each third subset of the initial training records and the second subset of the initial training records is different from each fourth subset of the initial training records.

7. The system of claim 6, wherein the operations further comprise updating the predictive model in response to receiving additional training data, including training the predictive model using the additional training data and a different training function than a training function previously used to train the predictive model.

8. The system of claim 6, wherein the predictive model is trained using a first training function and at least one of the one or more additional predictive models is trained using a second training function different from the first training function.

9. The system of claim 6, wherein identifying a predictive model based on the predictive request comprise identifying the predictive model based on a score for the predictive model and a respective score for each of the one or more additional predictive models, the score for each predictive model being based on a measure of effectiveness of the predictive model.

10. The system of claim 6, wherein the predictive request includes input data that is input to the predictive model to generate the result.

11. A non-transitory computer-readable storage medium encoded with a computer program, the program comprising instructions that when executed by a data processing apparatus cause the data processing apparatus to perform operations comprising:

receiving a predictive request;

identifying a predictive model based on the predictive request;

generating a result using the predictive model and the predictive request; and

generating a normalized result using a score normalization model and the result, the score normalization model being generated using intermediate training records output by the predictive model and intermediate training records output by one or more additional predictive models, wherein;

the predictive model is trained using a first subset of initial training records and the intermediate training records are generated by the predictive model using a second subset of the initial training records;

each additional predictive model is trained using a respective third subset of the initial training records and the additional training records output by the additional predictive model are generated by the additional predictive model using a respective fourth subset of the initial training records; and

the first subset of the initial training records is different from each third subset of the initial training records and the second subset of the initial training records is different from each fourth subset of the initial training records.

12. The non-transitory computer-readable storage medium of claim 11, wherein the operations further comprise updating the predictive model in response to receiving additional training data, including training the predictive model using the additional training data and a different training function than a training function previously used to train the predictive model.

13. The non-transitory computer-readable storage medium of claim 11, wherein the predictive model is trained using a first training function and at least one of the one or more additional predictive models is trained using a second training function different from the first training function.

14. The non-transitory computer-readable storage medium of claim 11, wherein identifying a predictive model based on the predictive request comprise identifying the predictive model based on a score for the predictive model and a respective score for each of the one or more additional predictive models, the score for each predictive model being based on a measure of effectiveness of the predictive model.