×

Target variable distribution-based acceptance of machine learning test data sets

  • US 10,726,356 B1
  • Filed: 08/01/2016
  • Issued: 07/28/2020
  • Est. Priority Date: 08/01/2016
  • Status: Active Grant
First Claim
Patent Images

1. A system, comprising:

  • one or more computing devices of a machine learning service of a provider network, wherein the one or more computing devices are configured to;

    identify, with respect to a particular machine learning model to be trained on behalf of a client to predict values of a target variable, a proposed training data set and a proposed test data set, wherein the target variable is an output variable of the particular machine learning model;

    determine that the proposed test data set meets a triggering criterion for execution of a selected target variable distribution comparison algorithm;

    obtain, based on an examination of at least a portion of the proposed training data set, a first statistical distribution of the target variable within the proposed training data set in accordance with the selected target variable distribution algorithm;

    obtain, based on an examination of at least a portion of the proposed test data set, a second statistical distribution of the target variable within the proposed test data set;

    compute a metric indicative of a difference between the first statistical distribution and the second statistical distribution;

    determine an acceptance criterion for evaluating the particular machine learning model, wherein said evaluating is to be performed after the particular machine learning model has been trained using the proposed training data set;

    determine, based at least in part on the metric, that the proposed test data set meets the acceptance criterion for evaluating the particular machine learning model; and

    provide, to the client, an indication of a prediction quality metric of the particular machine learning model, wherein the prediction quality metric is obtained using the proposed test data set.

View all claims
  • 1 Assignment
Timeline View
Assignment View
    ×
    ×