Normalization of predictive model scores

• US 8,370,279 B1
• Filed: 09/29/2011
• Issued: 02/05/2013
• Est. Priority Date: 09/29/2011
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Lin, Wei-Hao, Green, Travis H. K., Kaplow, Robert, Fu, Gang, Mann, Gideon S.
• Primary Examiner(s): Vincent, David R
• Assistant Examiner(s): Hill, Stanley K
• Application Number:
  US13/248,474
• Time in Patent Office: 495 Days
• Field of Search: None
• US Class Current: 706/12
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 5/02 Knowledge representation; Symbolic representation
• CPC Class Codes: G06N 5/022 Knowledge engineering; Knowledge acquisition

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for score normalization. One of the methods includes receiving initial training data, the initial training data comprising initial training records, each initial training record identifying input data as input and a category as output. The method includes generating a first trained predictive model using the initial training data and a training function. The method includes generating intermediate training records by inputting input data of the initial training records to a second trained predictive model, the second trained predictive model generated using the training function, each intermediate training record having a score. The method also includes generating a score normalization model using a score normalization training function and the intermediate training records.

Claims

1. A computer-implemented method comprising:

receiving initial training data, the initial training data comprising initial training records, each initial training record identifying input data as input and a category as output;

generating a trained predictive model using a first subset of the initial training data and a training function;

generating intermediate training records including inputting input data of a second subset of the initial training records to the trained predictive model, each intermediate training record having a score; and

generating a score normalization model using a score normalization training function and the intermediate training records.

2. The method of claim 1, further comprising:

dividing the initial training records into sets of training records, the sets of training records including a first set of training records;

generating a second trained predictive model using at least one of the sets of training records; and

generating an intermediate training record for each initial training record in the first set of training records, each intermediate training record having a value as input and the category of a respective initial training record as output, wherein the value is generated by inputting input data of the respective initial training record into the second trained predictive model.

3. The method of claim 2, wherein generating the second trained predictive model comprises generating the second trained predictive model using all of the sets of training records except the first set of training records.

4. The method of claim 1, further comprising:

calculating a mean of the scores of the intermediate training records; and

for each intermediate training record, reducing the score of the intermediate training record by the mean.

5. The method of claim 1, further comprising:

determining a high-end score for the scores of the intermediate training records;

identifying an intermediate training record having a score greater than the high-end score; and

changing the score of the identified intermediate training record to be the high-end score.

6. The method of claim 5, further comprising:

determining a low-end value for the scores of the intermediate training records;

identifying a second intermediate training record having a score less than the low-end score; and

changing the score of the identifier second intermediate training record to be the low-end score.

7. The method of claim 1 wherein the score normalization model is configured to generate scores associated with predictive outputs wherein the scores are between zero and one, inclusive.

8. The method of claim 1, further comprising:

receiving a predictive request from a client device, the predictive request including input data;

generating an intermediate output by inputting the input data to the trained predictive model;

generating a predictive output by providing the intermediate output to the score normalization model; and

providing the predictive output to the client device.

9. The method of claim 1, wherein each of the intermediate training records comprises a value for each distinct category in the initial training data.

10. A computer storage medium encoded with computer program instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:

receiving initial training data, the initial training data comprising initial training records, each initial training record identifying input data as input and a category as output;

generating a trained predictive model using the a first subset of initial training data and a training function;

generating intermediate training records including inputting input data of a second subset of the initial training records to the trained predictive model, each intermediate training record having a score; and

generating a score normalization model using a score normalization training function and the intermediate training records.

11. The medium of claim 10, further comprising:

dividing the initial training records into sets of training records, the sets of training records including a first set of training records;

generating a second trained predictive model using at least one of the sets of training records; and

generating an intermediate training record for each initial training record in the first set of training records, each intermediate training record having a value as input and the category of a respective initial training record as output, wherein the value is generated by inputting input data of the respective initial training record into the second trained predictive model.

12. The medium of claim 11, wherein generating the second trained predictive model comprises generating the second trained predictive model using all of the sets of training records except the first set of training records.

13. The medium of claim 10, further comprising:

calculating a mean of the scores of the intermediate training records; and

for each intermediate training record, reducing the score of the intermediate training record by the mean.

14. The medium of claim 10, further comprising instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:

determining a high-end score for the scores of the intermediate training records;

identifying an intermediate training record having a score greater than the high-end score; and

changing the score of the identified intermediate training record to be the high-end score.

15. The medium of claim 14, further comprising instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:

determining a low-end value for the scores of the intermediate training records;

identifying a second intermediate training record having a score less than the low-end score; and

changing the score of the identifier second intermediate training record to be the low-end score.

16. The medium of claim 10 wherein the score normalization model is configured to generate scores associated with predictive outputs wherein the scores are between zero and one, inclusive.

17. The medium of claim 10, further comprising instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:

receiving a predictive request from a client device, the predictive request including input data;

generating an intermediate output by inputting the input data to the first trained predictive model;

generating a predictive output by providing the intermediate output to the score normalization model; and

providing the predictive output to the client device.

18. The medium of claim 10, wherein each of the intermediate training records comprises a value for each distinct category in the initial training data.

19. A system comprising:

one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising;

receiving initial training data, the initial training data comprising initial training records, each initial training record identifying input data as input and a category as output;

generating a trained predictive model using a first subset of the initial training data and a training function;

generating intermediate training records including inputting input data of a second subset of the initial training records to the trained predictive model, each intermediate training record having a score; and

generating a score normalization model using a score normalization training function and the intermediate training records.

20. The system of claim 19 further comprising:

dividing the initial training records into sets of training records, the sets of training records including a first set of training records;

generating a second trained predictive model using at least one of the sets of training records; and

generating an intermediate training record for each initial training record in the first set of training records, each intermediate training record having a value as input and the category of a respective initial training record as output, wherein the value is generated by inputting input data of the respective initial training record into the second trained predictive model.

21. The system of claim 20, wherein generating the second trained predictive model comprises generating the second trained predictive model using all of the sets of training records except the first set of training records.

22. The system of claim 19, wherein the one or more storage devices further store instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:

calculating a mean of the scores of the intermediate training records; and

for each intermediate training record, reducing the score of the intermediate training record by the mean.

23. The system of claim 19, wherein the one or more storage devices further store instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:

determining a high-end score for the scores of the intermediate training records;

identifying an intermediate training record having a score greater than the high-end score; and

changing the score of the identified intermediate training record to be the high-end score.

24. The system of claim 23, wherein the one or more storage devices further store instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:

determining a low-end value for the scores of the intermediate training records;

identifying a second intermediate training record having a score less than the low-end score; and

changing the score of the identifier second intermediate training record to be the low-end score.

25. The system of claim 19 wherein the score normalization model is configured to generate scores associated with predictive outputs wherein the scores are between zero and one, inclusive.

26. The system of claim 19, wherein the one or more storage devices further store instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:

receiving a predictive request from a client device, the predictive request including input data;

generating an intermediate output by inputting the input data to the trained predictive model;

generating a predictive output by providing the intermediate output to the score normalization model; and

providing the predictive output to the client device.

27. The system of claim 19, wherein each of the intermediate training records comprises a value for each distinct category in the initial training data.