Normalization of predictive model scores

• US 9,406,019 B2
• Filed: 02/01/2013
• Issued: 08/02/2016
• Est. Priority Date: 09/29/2011
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Lin, Wei-Hao, Green, Travis H. K., Kaplow, Robert, Fu, Gang, Mann, Gideon S.
• Primary Examiner(s): Wong, Lut
• Application Number:
  US13/757,013
• Publication Number:
  US 20130144819A1
• Time in Patent Office: 1,278 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 5/02 Knowledge representation; Symbolic representation
• CPC Class Codes: G06N 5/022 Knowledge engineering; Knowledge acquisition

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for score normalization. One of the methods includes receiving initial training data, the initial training data comprising initial training records, each initial training record identifying input data as input and a category as output. The method includes generating a first trained predictive model using the initial training data and a training function. The method includes generating intermediate training records by inputting input data of the initial training records to a second trained predictive model, the second trained predictive model generated using the training function, each intermediate training record having a score. The method also includes generating a score normalization model using a score normalization training function and the intermediate training records.

Claims

1. A computer-implemented method comprising:

receiving initial training data, the initial training data comprising initial training records, each initial training record identifying input data as input and a category as output;

generating first intermediate training records by inputting input data of a first subset of the initial training records to a first trained predictive model, the first trained predictive model generated using at least a second subset of the initial training records and a training function, each first intermediate training record having a first score;

generating second intermediate training records by inputting input data of the second subset of the initial training records to a second trained predictive model, the second trained predictive model generated using the training function and at least the first subset of the initial training records, each second intermediate training record having a second score; and

generating, for the first trained predictive model and the second trained predictive model, a score normalization model using a score normalization training function, the first intermediate training records, and the second intermediate training records.

2. The method of claim 1, further comprising:

dividing the initial training records into subsets of initial training records, the subsets of initial training records including the first subset of the initial training records and the second subset of the initial training records;

generating the second trained predictive model using the first subset of the initial training records and at least one other subset of the initial training records of the subsets of training records.

3. The method of claim 2, wherein generating the second trained predictive model comprises generating the second trained predictive model using all of the subsets of initial training records except the subset of initial training records.

4. The method of claim 1, further comprising:

calculating a mean of the scores of the first and second intermediate training records; and

for each intermediate training record, reducing the score of the intermediate training record by the mean.

5. The method of claim 1, further comprising:

determining a high-end value for the scores of the first and second intermediate training records, the high-end value being a value at which a specified percent of scores of the first and second intermediate training records have a score that exceeds the high-end value;

identifying a first intermediate training record having a score greater than the high-end value; and

changing the score of the identified first intermediate training record to be the high-end value.

6. The method of claim 5, further comprising:

determining a low-end value for the scores of the first and second intermediate training records, the low-end value being a value at which a specified percent of scores of the first and second intermediate training records have a score that is less than the low-end value;

identifying a second intermediate training record having a score less than the low-end value; and

changing the score of the identified second intermediate training record to be the low-end value.

7. The method of claim 1 wherein the score normalization model is configured to generate scores associated with predictive outputs wherein the scores are between zero and one, inclusive.

8. The method of claim 1, further comprising:

receiving a predictive request from a client device, the predictive request including input data;

generating an intermediate output by inputting the input data to the first trained predictive model;

generating a predictive output by providing the intermediate output to the score normalization model; and

providing the predictive output to the client device.

9. The method of claim 1, wherein each of the intermediate training records comprises a value for each distinct category in the initial training data.

10. A non-transitory computer storage medium encoded with computer program instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:

receiving initial training data, the initial training data comprising initial training records, each initial training record identifying input data as input and a category as output;

generating first intermediate training records by inputting input data of a first subset of the initial training records to a first trained predictive model, the first trained predictive model generated using at least a second subset of the initial training records and a training function, each first intermediate training record having a first score;

generating second intermediate training records by inputting input data of the second subset of the initial training records to a second trained predictive model, the second trained predictive model generated using the training function and at least the first subset of the initial training records, each second intermediate training record having a second score; and

generating, for the first trained predictive model and the second trained predictive model, a score normalization model using a score normalization training function, the first intermediate training records, and the second intermediate training records.

11. The medium of claim 10, further comprising instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:

dividing the initial training records into subsets of initial training records, the subsets of initial training records including the first subset of the initial training records and the second subset of the initial training records;

generating the second trained predictive model using the first subset of the initial training records and at least one other subset of the initial training records of the subsets of training records.

12. The medium of claim 11, wherein generating the second trained predictive model comprises generating the second trained predictive model using all of the subsets of initial training records except the subset of initial training records.

13. The medium of claim 10, further comprising instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:

calculating a mean of the scores of the first and second intermediate training records; and

for each intermediate training record, reducing the score of the intermediate training record by the mean.

14. The medium of claim 10, further comprising instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:

determining a high-end value for the scores of the first and second intermediate training records, the high-end value being a value at which a specified percent of scores of the first and second intermediate training records have a score that exceeds the high-end value;

identifying a first intermediate training record having a score greater than the high-end value; and

changing the score of the identified first intermediate training record to be the high-end value.

15. The medium of claim 14, further comprising instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:

determining a low-end value for the scores of the first and second intermediate training records, the low-end value being a value at which a specified percent of scores of the first and second intermediate training records have a score that is less than the low-end value;

identifying a second intermediate training record having a score less than the low-end value; and

changing the score of the identified second intermediate training record to be the low-end value.

16. The medium of claim 10 wherein the score normalization model is configured to generate scores associated with predictive outputs wherein the scores are between zero and one, inclusive.

17. The medium of claim 10, further comprising instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:

receiving a predictive request from a client device, the predictive request including input data;

generating an intermediate output by inputting the input data to the first trained predictive model;

generating a predictive output by providing the intermediate output to the score normalization model; and

providing the predictive output to the client device.

18. The medium of claim 10, wherein each of the intermediate training records comprises a value for each distinct category in the initial training data.

19. A system comprising:

one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising;

receiving initial training data, the initial training data comprising initial training records, each initial training record identifying input data as input and a category as output;

generating first intermediate training records by inputting input data of a first subset of the initial training records to a first trained predictive model, the first trained predictive model generated using at least a second subset of the initial training records and a training function, each first intermediate training record having a first score;

generating second intermediate training records by inputting input data of the second subset of the initial training records to a second trained predictive model, the second trained predictive model generated using the training function and at least the first subset of the initial training records, each second intermediate training record having a second score; and

generating, for the first trained predictive model and the second trained predictive model, a score normalization model using a score normalization training function, the first intermediate training records, and the second intermediate training records.

20. The system of claim 19, wherein the one or more storage devices further store instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:

dividing the initial training records into subsets of initial training records, the subsets of initial training records including the first subset of the initial training records and the second subset of the initial training records;

generating the second trained predictive model using the first subset of the initial training records and at least one other subset of the initial training records of the subsets of training records.

21. The system of claim 20, wherein generating the second trained predictive model comprises generating the second trained predictive model using all of the subsets of initial training records except the subset of initial training records.

22. The system of claim 19, wherein the one or more storage devices further store instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:

calculating a mean of the scores of the first and second intermediate training records; and

for each intermediate training record, reducing the score of the intermediate training record by the mean.

23. The system of claim 19, wherein the one or more storage devices further store instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:

determining a high-end value for the scores of the first and second intermediate training records, the high-end value being a value at which a specified percent of scores of the first and second intermediate training records have a score that exceeds the high-end value;

identifying a first intermediate training record having a score greater than the high-end value; and

changing the score of the identified first intermediate training record to be the high-end value.

24. The system of claim 23, wherein the one or more storage devices further store instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:

determining a low-end value for the scores of the first and second intermediate training records, the low-end value being a value at which a specified percent of scores of the first and second intermediate training records have a score that is less than the low-end value;

identifying a second intermediate training record having a score less than the low-end value; and

changing the score of the identified second intermediate training record to be the low-end value.

25. The system of claim 19 wherein the score normalization model is configured to generate scores associated with predictive outputs wherein the scores are between zero and one, inclusive.

26. The system of claim 19, wherein the one or more storage devices further store instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:

receiving a predictive request from a client device, the predictive request including input data;

generating an intermediate output by inputting the input data to the first trained predictive model;

generating a predictive output by providing the intermediate output to the score normalization model; and

providing the predictive output to the client device.

27. The system of claim 19, wherein each of the intermediate training records comprises a value for each distinct category in the initial training data.