Predictive modeling accuracy

• US 8,843,427 B1
• Filed: 08/01/2011
• Issued: 09/23/2014
• Est. Priority Date: 07/01/2011
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Lin, Wei-Hao, Green, Travis H. K., Kaplow, Robert, Mann, Gideon S., Fu, Gang
• Primary Examiner(s): Gaffin, Jeffrey A
• Assistant Examiner(s): BROWN JR, NATHAN H
• Application Number:
  US13/195,349
• Time in Patent Office: 1,149 Days
• Field of Search: 706/45
• US Class Current: 706/45
• CPC Class Codes: G06N 20/00 Machine learning

Abstract

In general, a method includes receiving a training data set that includes a plurality of examples, wherein each example includes one or more features and an answer, generating a plurality of modified training data sets by applying one or more filters to the training data set, each of the plurality of modified training data sets being based on a different combination of the one or more filters, training a plurality of predictive models, each of the plurality of predictive models being trained using a different modified training data set of the plurality of modified training data sets, determining a respective accuracy for each of the plurality of predictive models, identifying a most accurate predictive model based on the determined accuracies, and specifying an association between the training data set and the combination of filters used to generate the modified training data set that was used to train the most accurate predictive model.

Claims

1. A computer-implemented method, the method comprising:

receiving a training data set that comprises a plurality of examples, wherein each example comprises one or more features and an answer;

generating a plurality of modified training data sets, wherein generating each of the modified training data sets comprises applying a respective filter or combination of filters to examples in the training data set to generate the examples of the modified training data set, the filter or combination of filters altering one or more of the examples;

training a plurality of predictive models, including training each of one or more predictive models using a different respective modified training data set of the plurality of modified training data sets;

determining a respective accuracy for each of the plurality of predictive models;

identifying a most accurate predictive model based on the determined respective accuracies; and

specifying an association between the training data set and the filter or combination of filters used to generate the modified training data set that was used to train the most accurate predictive model.

2. The computer-implemented method of claim 1, further comprising:

identifying one or more characteristics of the training data set.

3. The computer-implemented method of claim 2, wherein specifying an association between the training data set and the filter or combination of filters associated with the most accurate predictive model comprises specifying an association between the filter or combination of filters and the one or more characteristics of the training data set.

4. The computer-implemented method of claim 3, further comprising:

receiving a second training data set; and

determining that the second training data set has one or more characteristics that are similar to the one or more characteristics of the training data set.

5. The computer-implemented method of claim 4, wherein determining that the second training data set has one or more characteristics that are similar to the one or more characteristics of the training data set comprises:

determining that a level of similarity between the one or more characteristics of the second training data set and the one or more characteristics of the training data set satisfies a predetermined threshold.

6. The computer-implemented method of claim 4, further comprising applying the filter or combination of filters associated with the most accurate predictive model to the second training data set based on determining that the second training data set has one or more characteristics that are similar to the one or more characteristics of the training data set.

7. The computer-implemented method of claim 1, further comprising training an unfiltered predictive model based on the training data set, the unfiltered predictive model being trained without an application of a filter or combination of filters;

and

determining a level of accuracy associated with the unfiltered predictive model.

8. The computer-implemented method of claim 1, wherein:

the training data set comprises text-based data; and

the filter or combination of filters comprises one or more of an n-gram filter, a stop word filter, a punctuation filter, and a stemming filter.

9. The computer-implemented method of claim 1, wherein:

the training data set comprises numerical data; and

the filter or combination of filters comprises one or more of a shift and scale filter, a p-degree polynomial filter, and a spline transformation filter.

10. The computer-implemented method of claim 7, further comprising:

determining that the most accurate predictive model is the unfiltered predictive model;

identifying one or more characteristics of the training data set;

receiving a second training data set;

determining that the second training set has one or more characteristics that are similar to one or more characteristics of the training data set; and

training each of one or more predictive models using the second training data set without an application of a filter or combination of filters to the second training data set.

11. The computer-implemented method of claim 10, wherein determining that the second training set has one or more characteristics that are similar to the one or more characteristics of the training data set comprises:

determining that a level of similarity between the one or more characteristics of the second training data set and the one or more characteristics of the training data set satisfies a predetermined threshold.

12. One or more non-transitory computer storage devices comprising instructions that, when executed by one or more processing devices, cause the one or more processing devices to perform operations comprising:

receiving a training data set that comprises a plurality of examples, wherein each example comprises one or more features and an answer;

generating a plurality of modified training data sets, wherein generating each of the modified training data sets comprises applying a respective filter or combination of filters to examples in the training data set to generate the examples of the modified training data set, the filter or combination of filters altering one or more of the examples;

training a plurality of predictive models, including training each of one or more predictive models using a different respective modified training data set of the plurality of modified training data sets;

determining a respective accuracy for each of the plurality of predictive models;

identifying a most accurate predictive model based on the determined respective accuracies; and

specifying an association between the training data set and the filter or combination of filters used to generate the modified training data set that was used to train the most accurate predictive model.

13. The one or more computer storage devices of claim 12, further comprising instructions that, when executed by one or more processing devices, cause the one or more processing devices to perform operations comprising:

identifying one or more characteristics of the training data set.

14. The one or more computer storage devices of claim 13, wherein specifying an association between the training data set and the filter or combination of filters associated with the most accurate predictive model comprises specifying an association between the filter or combination of filters and the one or more characteristics of the training data set.

15. The one or more computer storage devices of claim 14, further comprising instructions that, when executed by one or more processing devices, cause the one or more processing devices to perform operations comprising:

receiving a second training data set; and

determining that the second training data set has one or more characteristics that are similar to the one or more characteristics of the training data set.

16. The one or more computer storage devices of claim 15, wherein determining that the second training data set has one or more characteristics that are similar to the one or more characteristics of the training data set comprises:

determining that a level of similarity between the one or more characteristics of the second training data set and the one or more characteristics of the training data set satisfies a predetermined threshold.

17. The one or more computer storage devices of claim 15, further comprising instructions that, when executed by one or more processing devices, cause the one or more processing devices to perform operations comprising:

applying the filter or combination of filters associated with the most accurate predictive model to the second training data set based on determining that the second training data set has one or more characteristics that are similar to the one or more characteristics of the training data set.

18. The one or more computer storage devices of claim 12, further comprising instructions that, when executed by one or more processing devices, cause the one or more processing devices to perform operations comprising:

training an unfiltered predictive model based on the training data set, the unfiltered predictive model being trained without an application of the one or more filters; and

determining a level of accuracy associated with the unfiltered predictive model.

19. The one or more computer storage devices of claim 12, wherein:

the training data set comprises text-based data; and

the filter or combination of filters comprises one or more of a n-gram filter, a stop word filter, a punctuation filter, or a stemming filter.

20. The one or more computer storage devices of claim 12, wherein:

the training data set comprises numerical data; and

the filter or combination of filters comprises one or more of a shift and scale filter, a p-degree polynomial filter, or a spline transformation filter.

21. The one or more computer storage devices of claim 18, further comprising:

determining that the most accurate predictive model is the unfiltered predictive model;

identifying one or more characteristics of the training data set;

receiving a second training data set;

determining that the second training set has one or more characteristics that are similar to one or more characteristics of the training data set; and

training each of one or more predictive models using the second training data set without an application of a filter or combination of filters to the second training data set.

22. The one or more computer storage devices of claim 21, wherein determining that the second training set has one or more characteristics that are similar to the one or more characteristics of the training data set comprises:

determining that a level of similarity between the one or more characteristics of the second training data set and the one or more characteristics of the training data set satisfies a predetermined threshold.

23. A system comprising:

one or more processing devices; and

one or more memory devices comprising instructions that, when executed by the one or more processing devices, cause the one or more processing devices to perform operations comprising;

receiving a training data set that comprises a plurality of examples, wherein each example comprises one or more features and an answer;

generating a plurality of modified training data sets, wherein generating each of the modified training data sets comprises applying a respective filter or combination of filters to examples in the training data set to generate the examples of the modified training data set, the filter or combination of filters altering one or more of the examples;

training a plurality of predictive models, including training each of one or more predictive models using a different respective modified training data set of the plurality of modified training data sets;

determining a respective accuracy for each of the plurality of predictive models;

identifying a most accurate predictive model based on the determined respective accuracies; and

specifying an association between the training data set and the filter or combination of filters used to generate the modified training data set that was used to train the most accurate predictive model.

24. The system of claim 23, wherein the one or more memory devices further comprise instructions that, when executed by one or more processing devices, cause the one or more processing devices to perform operations comprising:

identifying one or more characteristics of the training data set.

25. The system of claim 24, wherein specifying an association between the training data set and the filter or combination of filters associated with the most accurate predictive model comprises specifying an association between the filter or combination of filters and the one or more characteristics of the training data set.

26. The system of claim 25, wherein the one or more memory devices further comprise instructions that, when executed by one or more processing devices, cause the one or more processing devices to perform operations comprising:

receiving a second training data set; and

determining that the second training data set has one or more characteristics that are similar to the one or more characteristics of the training data set.

27. The system of claim 26, wherein determining that the second training data set has one or more characteristics that are similar to the one or more characteristics of the training data set comprises:

determining that a level of similarity between the one or more characteristics of the second training data set and the one or more characteristics of the training data set satisfies a predetermined threshold.

28. The system of claim 26, wherein the one or more memory devices further comprise instructions that, when executed by one or more processing devices, cause the one or more processing devices to perform operations comprising:

applying the filter or combination of filters associated with the most accurate predictive model to the second training data set based on determining that the second training data set has one or more characteristics that are similar to the one or more characteristics of the training data set.

29. The system of claim 23, wherein the one or more memory devices further comprise instructions that, when executed by one or more processing devices, cause the one or more processing devices to perform operations comprising:

training an unfiltered predictive model based on the training data set, the unfiltered predictive model being trained without an application of the one or more filters; and

determining a level of accuracy associated with the unfiltered predictive model.

30. The system of claim 23, wherein:

the training data set comprises text-based data; and

the filter or combination of filters comprises one or more of a n-gram filter, a stop word filter, a punctuation filter, and a stemming filter.

31. The system of claim 23, wherein:

the training data set comprises numerical data; and

the filter or combination of filters comprises one or more of a shift and scale filter, a p-degree polynomial filter, and a spline transformation filter.

32. The system of claim 29, wherein the one or more memory devices further comprises instructions that, when executed by one or more processing devices, cause the one or more processing devices to perform operations comprising:

determining that the most accurate predictive model is the unfiltered predictive model;

identifying one or more characteristics of the training data set;

receiving a second training data set;

determining that the second training set has one or more characteristics that are similar to one or more characteristics of the training data set; and

training each of one or more predictive models using the second training data set without an application of a filter or combination of filters to the second training data set.

33. The system of claim 32, wherein determining that the second training set has one or more characteristics that are similar to the one or more characteristics of the training data set comprises:

determining that a level of similarity between the one or more characteristics of the second training data set and the one or more characteristics of the training data set satisfies a predetermined threshold.

34. A computer-implemented method, the method comprising:

receiving a training data set that comprises a plurality of examples, wherein each example comprises one or more features and an answer;

identifying one or more characteristics of the training data set;

generating a plurality of modified training data sets by applying one or more filters to the training data set, each of the plurality of modified training data sets being based on a different combination of the one or more filters;

training a plurality of predictive models, each of the plurality of predictive models being trained using a different modified training data set of the plurality of modified training data sets;

determining a respective accuracy for each of the plurality of predictive models;

identifying a most accurate predictive model based on the determined respective accuracies; and

specifying an association between the training data set and the one or more filters used to generate the modified training data set that was used to train the most accurate predictive model, including specifying an association between the one or more filters and the one or more characteristics of the training data set.

35. The computer-implemented method of claim 34, further comprising:

receiving a second training data set; and

determining that the second training data set has one or more characteristics that are similar to the one or more characteristics of the training data set.

36. The computer-implemented method of claim 35, wherein determining that the second training data set has one or more characteristics that are similar comprises:

determining a level of similarity between the characteristics of the second training data set and the one or more characteristics of the training data set; and

determining that the level of similarity satisfies a threshold.

37. The computer-implemented method of claim 35, further comprising applying the one or more filters associated with the most accurate predictive model to the second training data set based on determining that the second training data set has one or more characteristics that are similar to the one or more characteristics of the training data set.

38. A computer-implemented method, the method comprising:

receiving a training data set that comprises a plurality of examples, wherein each example comprises one or more features and an answer;

generating a plurality of modified training data sets by applying one or more filters to the training data set, each of the plurality of modified training data sets being based on a different combination of the one or more filters;

training a plurality of predictive models, each of the plurality of predictive models being trained using a different modified training data set of the plurality of modified training data sets;

determining a respective accuracy for each of the plurality of predictive models;

training an unfiltered predictive model based on the training data set, the unfiltered predictive model being trained without an application of the one or more filters; and

determining a level of accuracy associated with the unfiltered predictive model

identifying a most accurate predictive model based on the determined respective accuracies; and

specifying an association between the training data set and the one or more filters used to generate the modified training data set that was used to train the most accurate predictive model.