Target variable distribution-based acceptance of machine learning test data sets
First Claim
Patent Images
1. A system, comprising:
- one or more computing devices of a machine learning service of a provider network, wherein the one or more computing devices are configured to;
identify, with respect to a particular machine learning model to be trained on behalf of a client to predict values of a target variable, a proposed training data set and a proposed test data set, wherein the target variable is an output variable of the particular machine learning model;
determine that the proposed test data set meets a triggering criterion for execution of a selected target variable distribution comparison algorithm;
obtain, based on an examination of at least a portion of the proposed training data set, a first statistical distribution of the target variable within the proposed training data set in accordance with the selected target variable distribution algorithm;
obtain, based on an examination of at least a portion of the proposed test data set, a second statistical distribution of the target variable within the proposed test data set;
compute a metric indicative of a difference between the first statistical distribution and the second statistical distribution;
determine an acceptance criterion for evaluating the particular machine learning model, wherein said evaluating is to be performed after the particular machine learning model has been trained using the proposed training data set;
determine, based at least in part on the metric, that the proposed test data set meets the acceptance criterion for evaluating the particular machine learning model; and
provide, to the client, an indication of a prediction quality metric of the particular machine learning model, wherein the prediction quality metric is obtained using the proposed test data set.
1 Assignment
0 Petitions
Accused Products
Abstract
Respective statistical distributions of a target variable within a proposed training data set and a proposed test data set for a machine learning model are obtained. A metric indicative of the difference between the two statistical distributions is computed. The difference metric is used to determine whether the proposed test data set is acceptable to evaluate the machine learning model.
-
Citations
21 Claims
-
1. A system, comprising:
one or more computing devices of a machine learning service of a provider network, wherein the one or more computing devices are configured to; identify, with respect to a particular machine learning model to be trained on behalf of a client to predict values of a target variable, a proposed training data set and a proposed test data set, wherein the target variable is an output variable of the particular machine learning model; determine that the proposed test data set meets a triggering criterion for execution of a selected target variable distribution comparison algorithm; obtain, based on an examination of at least a portion of the proposed training data set, a first statistical distribution of the target variable within the proposed training data set in accordance with the selected target variable distribution algorithm; obtain, based on an examination of at least a portion of the proposed test data set, a second statistical distribution of the target variable within the proposed test data set; compute a metric indicative of a difference between the first statistical distribution and the second statistical distribution; determine an acceptance criterion for evaluating the particular machine learning model, wherein said evaluating is to be performed after the particular machine learning model has been trained using the proposed training data set; determine, based at least in part on the metric, that the proposed test data set meets the acceptance criterion for evaluating the particular machine learning model; and provide, to the client, an indication of a prediction quality metric of the particular machine learning model, wherein the prediction quality metric is obtained using the proposed test data set. - View Dependent Claims (2, 3, 4, 5)
-
6. A method, comprising:
performing, by one or more computing devices; obtaining, based on an examination of at least a portion of a proposed training data set for a machine learning model, a first statistical distribution of a target variable within the proposed training data set, wherein the target variable is an output variable of the machine learning model, and wherein values of the target variable are to be predicted by the machine learning model; obtaining, based on an examination of at least a portion of a proposed test data set for the machine learning model, a second statistical distribution of the target variable within the proposed test data set; determining an acceptance criterion for evaluating the machine learning model; determining, based at least in part on a metric indicative of a difference between the first statistical distribution and the second statistical distribution, that the proposed test data set fails to meet the acceptance criterion for the machine learning model; and providing, via a programmatic interface, an indication that a test data set other than the proposed test data set should be utilized to evaluate the machine learning model. - View Dependent Claims (7, 8, 9, 10, 11, 12, 13, 14, 15)
-
16. A non-transitory computer-accessible storage medium storing program instructions that when executed on one or more processors:
-
identify (a) a proposed training data set for a machine learning model and (b) a proposed test data set for the machine learning model wherein values of a target variable are to be predicted by the machine learning model, and wherein the target variable is an output variable of the machine learning model; determine an acceptance criterion for evaluating the machine learning model; determine, based at least in part on a metric indicative of a difference between (a) a first statistical distribution of the target variable within the proposed training data set and (b) a second statistical distribution of the target variable within the proposed test data set, that the proposed test data set meets the acceptance criterion for the machine learning model; and provide, via a programmatic interface, an indication of approval of the proposed test data set for an evaluation of the machine learning model. - View Dependent Claims (17, 18, 19, 20, 21)
-
Specification