Transforming attributes for training automated modeling systems

• US 10,643,154 B2
• Filed: 09/21/2017
• Issued: 05/05/2020
• Est. Priority Date: 09/21/2016
• Status: Active Grant

• Current Assignee: Equifax, Inc.
• Original Assignee: Equifax, Inc.
• Inventors: Litherland, Trevis J., Hao, Li, Bondugula, Rajkumar
• Primary Examiner(s): Afshar, Kamran
• Assistant Examiner(s): Vaughn, Ryan C
• Application Number:
  US16/334,190
• Publication Number:
  US 20190205791A1
• Time in Patent Office: 957 Days
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 20/10 using kernel methods, e.g. support vector machines [SVM]
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06N 5/01 Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks

Abstract

In some aspects, a machine-learning model, which can transform input attribute values into a predictive or analytical output value, can be trained with training data grouped into attributes. A subset of the attributes can be selected and transformed into a transformed attribute used for training the model. The transformation can involve grouping portions of the training data for the subset of attributes into respective multi-dimensional bins. Each dimension of a multi-dimensional bin can correspond to a respective selected attribute. The transformation can also involve computing interim predictive output values. Each interim predictive output value can be generated from a respective training data portion in a respective multi-dimensional bin. The transformation can also involve computing smoothed interim output values by applying a smoothing function to the interim predictive output values. The transformation can also involve outputting the smoothed interim output values as a dataset for the transformed attribute.

Claims

1. A system comprising:

a processing device; and

one or more memory devices storing;

instructions executable by the processing device,

a machine-learning model that is a memory structure comprising input nodes interconnected with one or more output nodes via intermediate nodes, wherein the intermediate nodes are configured to transform input attribute values into a predictive or analytical output value for an entity associated with the input attribute values, and

training data for training the machine-learning model, wherein the training data are grouped into attributes;

wherein the processing device is configured to access the one or more memory devices and thereby execute the instructions to;

select a subset of attributes from the attributes of the training data;

transform the subset of attributes into a transformed attribute by performing operations comprising;

grouping (a) a first portion of the training data for the subset of attributes into a first multi-dimensional bin and (b) a second portion of the training data for the subset of attributes into a second multi-dimensional bin, wherein a dimension for each multi-dimensional bin corresponds to an attribute range of a respective one of the attributes in the subset of attributes,

computing a first set of interim predictive output values for a first attribute in the subset of attributes, wherein the first set of interim predictive output values is generated from a first subset of the training data within a first range of the attribute ranges,

computing a first set of smoothed interim output values by applying a smoothing function to the first set of interim predictive output values,

computing a second set of interim predictive output values for a second attribute in the subset of attributes, wherein the second set of interim predictive output values is generated from a second subset of the training data within a second range of the attribute ranges,

computing a second set of smoothed interim output values by applying the smoothing function to the second set of interim predictive output values, and

outputting a dataset for the transformed attribute, the dataset having, at least, a first dimension including the first set of smoothed interim output values and a second dimension including the second set of smoothed interim output values; and

train the machine-learning model with the transformed attribute.

2. The system of claim 1, wherein the smoothing function comprises an isotonic regression formula.

3. The system of claim 1, wherein the processing device is further configured to execute the instructions to receive a monotonicity constraint that is applied to the machine-learning model, wherein one or more attributes in the subset of attributes violate the monotonicity constraint, and wherein the smoothing function is configured to cause the smoothed interim predictive output values to comply with the monotonicity constraint.

4. The system of claim 1, wherein the processing device is configured to select the subset of attributes by performing operations comprising, for a particular attribute from the subset of attributes:

retrieving, from the training data, data samples including values for the particular attribute and corresponding values for a response variable;

computing a correlation between the data samples having the values for the particular attribute and the corresponding values for the response variable; and

including the particular attribute in the subset of attributes based on the correlation exceeding a threshold correlation indicative of monotonicity between the particular attribute and the response variable.

5. The system of claim 1, wherein the processing device is configured to exclude a particular attribute from the subset of attributes by performing operations comprising:

identifying a time period having a plurality of intervals;

determining that the particular attribute lacks data values for a subset of intervals from the plurality of intervals; and

excluding the particular attribute from the subset of attributes based on the subset of intervals having missing data values exceeding a threshold number of interval.

6. The system of claim 1, wherein the processing device is configured to select the subset of attributes by performing operations comprising, for a first attribute and a second attribute from the subset of attributes:

retrieving, from the training data, first data samples including values for the first attribute and second data samples including values for the second attribute;

computing a correlation between the first data samples and the second data samples; and

including the first attribute and the second attribute in the subset of attributes based on the correlation being less than a threshold correlation indicative of multicollinearity between the first attribute and the second attribute.

7. The system of claim 1, wherein the processing device is configured to compute a particular interim predictive output value for a particular multi-dimensional bin by performing operations comprising:

computing, for combinations of attribute values in the particular multi-dimensional bin, respective probabilities of an outcome; and

averaging the probabilities to obtain the particular interim predictive output value.

8. A method comprising:

accessing, from a non-transitory computer-readable medium, (i) a machine-learning model that transforms input attribute values into a predictive or analytical output value for an entity associated with the input attribute values and (ii) training data for training the machine-learning model, wherein the training data are grouped into attributes;

selecting, by a processing device, a subset of attributes from the attributes of the training data;

transforming, by the processing device, the subset of attributes into a transformed attribute by performing operations comprising;

grouping (a) a first portion of the training data for the subset of attributes into a first multi-dimensional bin and (b) a second portion of the training data for the subset of attributes into a second multi-dimension bin wherein a dimension for each multi-dimensional bin corresponds to an attribute range of a respective on of the attributes in the subset of attributes,

computing a first set of interim predictive output values for a first attribute in the subset of attributes, wherein the first set of interim predictive output values is generated from a first subset of the training data within a first range of the attribute ranges,

computing a first set of smoothed interim output values by applying a smoothing function to the first set of interim predictive output values,

computing a second set of interim predictive output values for a second attribute in the subset of attributes, wherein the second set of interim predictive output values is generated from a second subset of the training data within a second range of the attribute ranges,

computing a second set of smoothed interim output values by applying the smoothing function to the second set of interim predictive output values, and

outputting a dataset for the transformed attribute, the dataset having, at least, a first dimension including the first set of smoothed interim output values and a second dimension including the second set of smoothed interim output values; and

training, by the processing device, the machine-learning model with the transformed attribute.

9. The method of claim 8, wherein the smoothing function comprises an isotonic regression formula.

10. The method of claim 9, further comprising receiving a monotonicity constraint that is applied to the machine-learning model, wherein one or more attributes in the subset of attributes violates the monotonicity constraint, and wherein applying the smoothing function causes the smoothed interim predictive output values to comply with the monotonicity constraint.

11. The method of claim 8, wherein selecting the subset of attributes comprises performing, by the processing device and for a particular attribute from the subset of attributes, operations comprising:

retrieving, from the training data, data samples including values for the particular attribute and corresponding values for a response variable;

computing a correlation between the data samples having the values for the particular attribute and the corresponding values for the response variable; and

including the particular attribute in the subset of attributes based on the correlation exceeding a threshold correlation indicative of monotonicity between the particular attribute and the response variable.

12. The method of claim 8, further comprising excluding, by the processing device, a particular attribute from the subset of attributes by performing operations comprising:

identifying a time period having a plurality of intervals;

determining that the particular attribute lacks data values for a subset of intervals from the plurality of intervals; and

excluding the particular attribute from the subset of attributes based on the subset of intervals having missing data values exceeding a threshold number of intervals.

13. The method of claim 8, wherein selecting the subset of attributes comprises performing, by the processing device and for a first attribute and a second attribute from the subset of attributes, operations comprising:

retrieving, from the training data, first data samples including values for the first attribute and second data samples including values for the second attribute;

computing a correlation between the first data samples and the second data samples; and

including the first attribute and the second attribute in the subset of attributes based on the correlation being less than a threshold correlation indicative of multicollinearity between the first attribute and the second attribute.

14. The method of claim 8, further comprising computing, by the processing device, a particular interim predictive output value for a particular multi-dimensional bin by performing operations comprising:

computing, for combinations of attribute values in the particular multi-dimensional bin, respective probabilities of an outcome; and

averaging the probabilities to obtain the particular interim predictive output value.

15. A non-transitory computer-readable medium in which instructions executable by a processing device are stored for causing the processing device to:

access (i) a machine-learning model that is a memory structure comprising input nodes interconnected with one or more output nodes via intermediate nodes, wherein the intermediate nodes are configured to transform input attribute values into a predictive or analytical output value for an entity associated with the input attribute values and (ii) training data for training the machine-learning model, wherein the training data are grouped into attributes;

select a subset of attributes from the attributes of the training data;

transform the subset of attributes into a transformed attribute by performing operations comprising;

grouping (a) a first portion of the training data for the subset of attributes into a first multi-dimensional bin and (b) a second portion of the training data for the subset of attributes into a second multi-dimension bin, wherein a dimension for each multi-dimensional bin corresponds to an attribute range of a respective one of the attributes in the subset of attributes,

computing a first set of interim predictive output values for a first attribute in the subset of attributes, wherein the first set of interim predictive output values is generated from a first subset of the training data within a first range of the attribute ranges,

computing a first set of smoothed interim output values by applying a smoothing function to the first set of interim predictive output values,

computing a second set of interim predictive output values for a second attribute in the subset of attributes, wherein the second subset of interim predictive output values is generated from a second subset of the training data within a second range of the attribute ranges,

computing a second set of smoothed interim output values by applying the smoothing function to the second set of interim predictive output values, and

outputting a dataset for the transformed attribute, the dataset having, at least, a first dimension including the first set of smoothed interim output values and a second dimension including the second set of smoothed interim output values; and

train the machine-learning model with the transformed attribute.

16. The non-transitory computer-readable medium of claim 15, wherein the smoothing function comprises an isotonic regression formula.

17. The non-transitory computer-readable medium of claim 16, wherein the instructions, when executed the processing device, cause the processing device to receive a monotonicity constraint that is applied to the machine-learning model, wherein one or more attributes in the subset of attributes violate the monotonicity constraint, and wherein the smoothing function is configured to cause the smoothed interim predictive output values to comply with the monotonicity constraint.

18. The non-transitory computer-readable medium of claim 15, wherein the instructions, when executed the processing device, cause the processing device to select the subset of attributes by performing operations comprising, for a particular attribute from the subset of attributes:

retrieving, from the training data, data samples including values for the particular attribute and corresponding values for a response variable;

computing a correlation between the data samples having the values for the particular attribute and the corresponding values for the response variable; and

including the particular attribute in the subset of attributes based on the correlation exceeding a threshold correlation indicative of monotonicity between the particular attribute and the response variable.

19. The non-transitory computer-readable medium of claim 15, wherein the instructions, when executed the processing device, cause the processing device to select the subset of attributes by performing operations comprising, for a first attribute and a second attribute from the subset of attributes:

retrieving, from the training data, first data samples including values for the first attribute and second data samples including values for the second attribute;

computing a correlation between the first data samples and the second data samples; and

including the first attribute and the second attribute in the subset of attributes based on the correlation being less than a threshold correlation indicative of multicollinearity between the first attribute and the second attribute.

20. The non-transitory computer-readable medium of claim 15, wherein the instructions, when executed the processing device, cause the processing device to compute a particular interim predictive output value for a particular multi-dimensional bin by performing operations comprising:

computing, for combinations of attribute values in the particular multi-dimensional bin, respective probabilities of an outcome; and

averaging the probabilities to obtain the particular interim predictive output value.