Feature generation and model selection for generalized linear models

• US 9,292,550 B2
• Filed: 02/21/2013
• Issued: 03/22/2016
• Est. Priority Date: 02/21/2013
• Status: Active Grant

• Current Assignee: Oracle International Corporation (Oracle Corporation)
• Original Assignee: Oracle International Corporation (Oracle Corporation)
• Inventors: Yarmus, Joseph
• Primary Examiner(s): Beausoliel, Jr., Robert
• Assistant Examiner(s): Pham, Michael
• Application Number:
  US13/772,852
• Publication Number:
  US 20140236965A1
• Time in Patent Office: 1,125 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 16/22 Indexing; Data structures therefor; Storage structures

Abstract

Systems, methods, and other embodiments associated with feature generation and model selection for generalized linear models are described. In one embodiment, a method includes ordering candidate features in a dataset being considered by a streamwise feature selection process according to an inclusion score that reflects a likelihood that a given candidate feature will be included in the GLM. The ordered candidate features are provided to the streamwise feature selection process for acceptance testing. In one embodiment, the method also includes selecting penalty criterion for use in the acceptance testing that is based on characteristics of the dataset.

Claims

1. A computer-implemented method comprising:

identifying a dataset that stores values for a target attribute and input attributes, where the input attributes are under consideration for inclusion in a generalized linear model that predicts a value of the target attribute based on a selection of features, where each feature comprises a combination of one or more of the input attributes;

identifying candidate features, where a candidate feature comprises a combination of one or more of the input attributes;

computing respective inclusion scores for respective candidate features, based, at least in part on a likelihood that the candidate feature will be selected for inclusion in the generalized linear model;

ordering the candidate features according to inclusion score;

constructing a set of one or more branches of candidate features, where each branch includes candidate features ordered according to inclusion score from highest inclusion score to lowest inclusion score, where the one or more branches do not include candidate features having an inclusion score below a predetermined minimum score; and

providing a branch of candidate features to a streamwise feature selection process configured to construct the generalized linear model by considering candidate features in the branch, in turn, starting with the candidate feature with the highest inclusion score, and including selected candidate features in the generalized linear model.

2. The computer-implemented method of claim 1, comprising computing the inclusion scores for a branch in a single scan of the input attributes.

3. The computer-implemented method of claim 1, comprising providing one or more termination criteria to the streamwise feature selection process such that when the termination criteria are met, consideration of candidate features in a branch by the streamwise feature selection process is terminated, where the termination criteria are based, at least in part, on one or more of:

a failure to accept a candidate feature having a highest inclusion score;

a predetermined number of consecutive failures to accept candidate features having next highest inclusion scores; and

when no remaining candidate features have a sufficiently high inclusion score, such that a lower bound on a penalized cost of a candidate model is less than a penalized cost of a last accepted model.

4. The computer-implemented method of claim 1, comprising:

determining that one or more candidate features in a branch have been accepted for inclusion in the generalized linear model;

computing updated inclusion scores for remaining candidate features in the branch that have not yet been provided to the streamwise feature selection process;

where the updated inclusion scores are computed based, at least in part, on a correlation between respective candidate features and a residual error of a last accepted generalized linear model that includes the one or more accepted candidate features; and

providing the remaining candidate features to the streamwise feature selection process in order of updated inclusion scores.

5. The computer-implemented method of claim 1, further comprising

upon termination by the streamwise feature selection process of consideration of candidate features in the branch, identifying candidate features that were accepted by the streamwise selection process, and constructing subsequent branches by:

generating one or more branches of candidate features using remaining input attributes that have not been considered for inclusion in a branch, or

generating a set of one or more second order branches of new candidate features by combining, for each branch in the set of second order branches, an accepted candidate feature with the input attributes;

computing respective inclusion scores for respective new candidate features in the subsequent branch, based, at least in part on a likelihood that the new candidate feature will be selected for inclusion in the generalized linear model;

constructing the subsequent branch of new candidate features ordered according to inclusion score; and

providing the subsequent branch of candidate features, in order of inclusion score, to the streamwise feature selection process.

6. The computer-implemented method of claim 5, where candidate features in the second order branches comprise respective candidate features that are the product of an accepted candidate feature and respective input attributes.

7. The computer-implemented method of claim 6, comprising:

upon termination by the streamwise feature selection process of the second order branches, constructing a set of one or more third order branches by;

identifying second order candidate features that were accepted by the streamwise selection process;

generating branches of respective third order candidate features by combining, for each third order branch, an accepted second order candidate feature with respective input attributes;

computing respective inclusion scores for respective third order candidate features in the subsequent branch, based, at least in part on a likelihood that the third order candidate feature will be selected for inclusion in the generalized linear model;

constructing the third order branch of new candidate features ordered according to inclusion score; and

providing the third order branch of candidate features, in order of inclusion score, to the streamwise feature selection process.

8. The computer-implemented method of claim 5, where respective inclusion scores are based, at least in part, on a correlation between respective candidate features and i) residual errors a last accepted generalized linear model or ii) last iteration working residuals of a last accepted generalized linear model.

9. The computer-implemented method of claim 6, further comprising terminating the streamwise feature selection process when all branches meet one or more termination criteria.

10. The computer-implemented method of claim 1, where the streamwise feature selection process performs acceptance testing to select candidate features by:

constructing a candidate generalized linear model that includes a next candidate feature in the branch;

comparing the candidate generalized linear models to a last accepted generalized linear model;

accepting the candidate generalized linear model when acceptance criteria are met, where the acceptance criteria are based, at least in part, on a penalty criterion;

further where the penalty criterion is selected, at least in part, on characteristics of the dataset; and

where the streamwise feature selection process uses the penalty criterion to compute a penalized cost of a candidate generalized linear model and accepts the candidate generalized linear model when the penalized cost is less than a penalized cost for the last accepted generalized linear model.

11. The computer-implemented method of claim 10 where the characteristics of the dataset comprise one or more of:

a number of rows and a number of input attributes.

12. The computer-implemented method of claim 1, further comprising:

partitioning the dataset into a first partition and a second partition;

computing inclusion scores using input attribute values and target attribute values in the first partition; and

further where the streamwise feature selection process tests candidate features for inclusion in the generalized linear model using input attribute values and target attribute values in the second partition.

13. The computer-implemented method of claim 1, comprising, when the target attribute is numeric, calculating an initial inclusion score for a candidate feature based on a correlation of the candidate feature to the target attribute.

14. The computer-implemented method of claim 1, comprising, when the target attribute is categorical, calculating an initial inclusion score for a candidate feature based on a t-statistic computed from a sample difference in values of the candidate feature by target attribute class.

15. The computer-implemented method of claim 1, further comprising:

identifying a maximum branch size, where the branch size is a fixed value, less than or equal to the number of input attributes in the dataset; and

limiting a size of branches to the maximum branch size.

16. The computer-implemented method of claim 1, further comprising constructing a candidate dataset by storing, in cache memory:

a portion of the dataset corresponding to input attribute values that are combined in candidate features included in a last accepted generalized linear model; and

a portion of the dataset corresponding to input attribute values that are combined in a next candidate feature in the branch.

17. The computer-implemented method of claim 16, further comprising constructing the candidate dataset by storing, in cache memory, a portion of the dataset corresponding to input attribute values that are combined in a set of candidate features having highest inclusion scores.

18. The computer-implemented method of claim 1, comprising:

identifying sparse input attributes in the dataset;

computing respective inclusion scores for respective sparse input attributes;

creating an ordered list of sparse input attributes having an inclusion score higher than a threshold sparse input attribute inclusion score;

choosing a set of k sparse input attributes, by sampling the ordered list, where the sampling of respective sparse input attributes is weighted according to respective inclusion scores;

constructing a candidate feature whose row value instance is a sum of respective inclusion scores for respective k sparse input attributes multiplied by respective sparse input attribute values; and

adding the candidate feature to the branch.

19. The computer-implemented method of claim 18, further comprising:

choosing a subsequent set of k sparse input attributes, by sampling the ordered list, where the sampling of respective sparse input attributes is weighted according to a sampling weight comprising respective inclusion scores reduced by a predetermined factor;

constructing a candidate feature whose row value instance is a sum of respective inclusion scores for respective k sparse input attributes multiplied by respective sparse input attribute values; and

adding the candidate feature to the branch.

20. A computing system, comprising:

a branch construction logic configured to;

identify a dataset that stores values for a target attribute and input attributes identifying candidate features, where a candidate feature comprises a combination of one or more input attributes;

compute respective inclusion scores for respective candidate features in the branch, based, at least in part on a likelihood that the candidate feature will be selected for inclusion in a generalized linear model that combines selected candidate features to predict a value of the target attribute;

order the candidate features according to inclusion score; and

construct a branch of candidate features ordered according to inclusion score from highest inclusion score to lowest inclusion score, where the branch does not include candidate features having an inclusion score below a predetermined minimum score; and

a streamwise feature selection logic configure to construct the generalized linear model by performing acceptance testing on candidate generalized linear models that include a next candidate feature in the branch, where candidate generalized linear models are compared to a last accepted generalized linear model and a candidate generalized linear model is accepted when one or more acceptance criteria are met.

21. The computing system of claim 20, where the branch construction logic is configured to:

determine that one or more candidate features in a branch have been accepted for inclusion in the generalized linear model;

compute updated inclusion scores for remaining candidate features in the branch that have not yet been provided to the streamwise feature selection process;

where the updated inclusion scores are computed based, at least in part, on a correlation between respective candidate features and a residual error of the last accepted generalized linear model that includes the one or more accepted candidate features; and

re-order the remaining candidate features in the branch according to updated inclusion scores.

22. The computing system of claim 20, further comprising:

an acceptance criteria logic configured to select the acceptance criteria based, at least in part, on a penalty criterion, where the acceptance criteria logic is configured to select the penalty criterion, at least in part, on characteristics of the dataset; and

where the streamwise feature selection logic is configured to use the penalty criterion to compute a penalized cost of a candidate generalized linear model and accept the candidate generalized linear model when the penalized cost is less than a penalized cost for the last accepted generalized linear model.

23. The computing system of claim 20, further comprising a dataset construction logic configured to construct a working dataset by storing, in cache memory, one or more of:

a portion of the dataset corresponding to input attribute values that are combined in candidate features included the last accepted generalized linear model; and

a portion of the dataset corresponding to input attribute values that are combined in a next candidate feature in the branch; and

a portion of the dataset corresponding to input attribute values that are combined in a set of candidate features having highest inclusion scores.

24. A non-transitory computer-readable medium storing computer-executable instructions that when executed by a computer cause the computer to at least:

identify a dataset that stores values for one or more input attributes and a target attribute;

until one or more model termination criteria are met, construct a set of branches, where each branch comprises candidate features ordered according to respective inclusion score, where a candidate features comprises a combination of one or more input attributes, further where respective inclusion scores estimate a likelihood that respective candidate features will be selected for inclusion in a generalized linear model that comprises a combination of features that predict the target attribute;

until one or more branch termination criteria are met for each branch in the set of branches;

construct a candidate generalized linear model that includes a next candidate feature in the branch;

perform acceptance testing on the candidate generalized linear model such that when the candidate generalized linear model meets acceptance criteria, the candidate generalized linear model is accepted;

when one or more reorder criteria are met, compute updated inclusion scores for remaining candidate features in the branch based, at least in part, on a correlation between respective candidate features and a residual error of the last accepted generalized linear model;

re-order the remaining candidate features in the branch according to updated inclusion scores; and

when branch termination criteria are met for the branch, access a next branch in the set of branches; and

provide the last accepted generalized linear model as an output.

25. The non-transitory computer-readable medium of claim 24, where the acceptance criteria are based, at least in part, on a penalty criterion, further where the instructions further comprise instructions configured to cause the computer to:

select the penalty criterion, at least in part, on a number of rows or a number of attributes in the dataset;

use the penalty criterion to compute a penalized cost of a candidate generalized linear model; and

accept the candidate generalized linear model when the penalized cost is less than a penalized cost for the last accepted generalized linear model.

26. The non-transitory computer-readable medium of claim 24, where the instructions comprise instructions configured to cause the computer to compute inclusion scores for candidate features using data in a first data partition and perform acceptance testing on the candidate generalized linear model using data in a second data partition.

27. The non-transitory computer-readable medium of claim 24, where the branch termination criteria comprise one or more of:

a failure to accept a candidate feature having a highest inclusion score;

a predetermined number of consecutive failures to accept candidate features having next highest inclusion scores; and

when no remaining candidates have a sufficiently high inclusion score.

28. The non-transitory computer-readable medium of claim 24, where the reorder criteria comprise a predetermined reduction in model error when a most recent candidate generalized linear model is accepted.

29. The non-transitory computer-readable medium of claim 24, where the model termination criteria comprise terminating when branch termination criteria have been met for all branches.

30. A non-transitory computer-readable medium storing computer-executable instructions that when executed by a computer cause the computer to at least:

identify a dataset that stores values for a target attribute and input attributes, where the input attributes are under consideration for inclusion in a generalized linear model that predicts a value of the target attribute based on a selection of features, where each feature comprises a combination of one or more of the input attributes;

identify candidate features, where a candidate feature comprises a combination of one or more of the input attributes;

compute respective inclusion scores for respective candidate features, based, at least in part on a likelihood that the candidate feature will be selected for inclusion in the generalized linear model;

order the candidate features according to inclusion score;

construct a set of one or more branches of candidate features, where each branch includes candidate features ordered according to inclusion score from highest inclusion score to lowest inclusion score, where the one or more branches do not include candidate features having an inclusion score below a predetermined minimum score; and

provide a branch of candidate features to a streamwise feature selection process configured to construct the generalized linear model by considering candidate features in the branch, in turn, starting with the candidate feature with the highest inclusion score, and including selected candidate features in the generalized linear model.

31. The non-transitory computer-readable medium of claim 30, where the instructions include instructions configured to cause the computer to provide one or more termination criteria to the streamwise feature selection process such that when the termination criteria are met, consideration of candidate features in a branch by the streamwise feature selection process is terminated, where the termination criteria are based, at least in part, on one or more of:

a failure to accept a candidate feature having a highest inclusion score;

a predetermined number of consecutive failures to accept candidate features having next highest inclusion scores; and

when no remaining candidate features have a sufficiently high inclusion score, such that a lower bound on a penalized cost of a candidate model is less than a penalized cost of a last accepted model.

32. The non-transitory computer-readable medium of claim 30, where the instructions comprise instructions configured to cause the computer to:

determine that one or more candidate features in a branch have been accepted for inclusion in the generalized linear model;

compute updated inclusion scores for remaining candidate features in the branch that have not yet been provided to the streamwise feature selection process;

where the updated inclusion scores are computed based, at least in part, on a correlation between respective candidate features and a residual error of a last accepted generalized linear model that includes the one or more accepted candidate features; and

provide the remaining candidate features to the streamwise feature selection process in order of updated inclusion scores.

33. A computer-implemented method, comprising:

identifying a dataset that stores values for one or more input attributes and a target attribute;

until one or more model termination criteria are met, constructing a set of branches, where each branch comprises candidate features ordered according to respective inclusion score, where a candidate features comprises a combination of one or more input attributes, further where respective inclusion scores estimate a likelihood that respective candidate features will be selected for inclusion in a generalized linear model that comprises a combination of features that predict the target attribute;

until one or more branch termination criteria are met for each branch in the set of branches;

constructing a candidate generalized linear model that includes a next candidate feature in the branch;

performing acceptance testing on the candidate generalized linear model such that when the candidate generalized linear model meets acceptance criteria, the candidate generalized linear model is accepted;

when one or more reorder criteria are met, computing updated inclusion scores for remaining candidate features in the branch based, at least in part, on a correlation between respective candidate features and a residual error of the last accepted generalized linear model;

re-ordering the remaining candidate features in the branch according to updated inclusion scores; and

when branch termination criteria are met for the branch, accessing a next branch in the set of branches; and

providing the last accepted generalized linear model as an output.

34. The computer-implemented method of claim 33, where the acceptance criteria are based, at least in part, on a penalty criterion, where the method further comprises:

selecting the penalty criterion, at least in part, on a number of rows or a number of attributes in the dataset;

using the penalty criterion to compute a penalized cost of a candidate generalized linear model; and

accepting the candidate generalized linear model when the penalized cost is less than a penalized cost for the last accepted generalized linear model.

35. The computer-implemented method of claim 33, further comprising computing inclusion scores for candidate features using data in a first data partition and performing acceptance testing on the candidate generalized linear model using data in a second data partition.

36. The computer-implemented method of claim 33, where the branch termination criteria comprise one or more of:

a failure to accept a candidate feature having a highest inclusion score;

a predetermined number of consecutive failures to accept candidate features having next highest inclusion scores; and

when no remaining candidates have a sufficiently high inclusion score.

37. The computer-implemented method of claim 33, where the reorder criteria comprise a predetermined reduction in model error when a most recent candidate generalized linear model is accepted.

38. The computer-implemented method of claim 33, where the model termination criteria comprise terminating when branch termination criteria have been met for all branches.