Feature processing tradeoff management

• US 10,169,715 B2
• Filed: 09/17/2014
• Issued: 01/01/2019
• Est. Priority Date: 06/30/2014
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Dirac, Leo Parker, Correa, Nicolle M., Dannaker, Charles Eric
• Primary Examiner(s): Smith, Paulinho E
• Application Number:
  US14/489,448
• Publication Number:
  US 20150379427A1
• Time in Patent Office: 1,567 Days
• Field of Search: None
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46

Abstract

At a machine learning service, a set of candidate variables that can be used to train a model is identified, including at least one processed variable produced by a feature processing transformation. A cost estimate indicative of an effect of implementing the feature processing transformation on a performance metric associated with a prediction goal of the model is determined. Based at least in part on the cost estimate, a feature processing proposal that excludes the feature processing transformation is implemented.

Claims

1. A system, comprising:

one or more computing devices configured to;

determine, via one or more programmatic interactions with a client of a machine learning service of a provider network, (a) one or more target variables to be predicted using a specified training data set, (b) one or more prediction quality metrics including a particular prediction quality metric, and (c) one or more prediction run-time goals including a particular prediction run-time goal;

identify a set of candidate feature processing transformations to derive a first set of processed variables from one or more input variables of the specified data set, wherein at least a subset of the first set of processed variables is usable to train a machine learning model to predict the one or more target variables, and wherein the set of candidate feature processing transformations includes a particular feature processing transformation;

determine (a) a quality estimate indicative of an effect, on the particular prediction quality metric, of implementing the particular candidate feature processing transformation, and (b) a cost estimate indicative of an effect, on a particular run-time performance metric associated with the particular prediction run-time goal, of implementing the particular candidate feature processing transformation;

generate, based at least in part on the quality estimate and at least in part on the cost estimate, a feature processing proposal to be provided to the client for approval, wherein the feature processing proposal includes a recommendation to implement the particular feature processing transformation; and

in response to an indication of approval from the client, execute a machine learning model trained using a particular processed variable obtained from the particular feature processing transformation.

2. The system as recited in claim 1, wherein to determine the quality estimate, the one or more computing devices implement a plurality of evaluation runs of the machine learning model, including a first evaluation run in which a first set of values of the particular processed variable are provided as input to the machine learning model, and a second evaluation run in which a different set of values of the particular processed variable are provided as input to the machine learning model.

3. The system as recited in claim 1, wherein to determine the cost estimate, the one or more computing devices implement respective evaluation runs of a first variant of the machine learning model and a second variant of the machine learning model, wherein the first variant is trained using a first training set that includes the particular processed variable, and the second variant is trained using a second training set that excludes the particular processed variable.

4. The system as recited in claim 1, wherein the particular prediction quality metric comprises one or more of:

(a) an AUC (area under curve) metric, (b) an accuracy metric, (c) a recall metric, (d) a sensitivity metric, (e) a true positive rate, (f) a specificity metric, (g) a true negative rate, (h) a precision metric, (i) a false positive rate, (j) a false negative rate, (k) an F1 score, (l) a coverage metric, (m) an absolute percentage error metric, or (n) a squared error metric.

5. The system as recited in claim 1, wherein the particular feature processing transformation comprises a use of one or more of:

(a) a quantile bin function, (b) a Cartesian product function, (c) a bi-gram function, (d) an n-gram function, (e) an orthogonal sparse bigram function, (f) a calendar function, (g) an image processing function, (h) an audio processing function, (i) a bio-informatics processing function, or (j) a natural language processing function.

6. A method, comprising:

performing, by one or more computing devices;

identifying, at a machine learning service, a set of candidate input variables usable to train a machine learning model to predict one or more target variables, wherein the set of candidate input variables includes at least a particular processed variable generated by a particular feature processing transformation applicable to one or more input variables of a training data set;

determining (a) a quality estimate indicative of an effect, on a particular prediction quality metric, of implementing the particular feature processing transformation, and (b) a cost estimate indicative of an effect, on a performance metric associated with a particular prediction goal, of implementing the particular feature processing transformation; and

implementing, based at least in part on the quality estimate and at least in part on the cost estimate, a feature processing plan that includes the particular feature processing transformation.

7. The method as recited in claim 6, further comprising performing, by the one or more computing devices:

generating one or more feature processing proposals, including a particular feature processing proposal recommending the particular feature processing transformation, based at least in part on an analysis of respective quality estimates and respective cost estimates corresponding to a plurality of candidate feature processing transformations; and

providing an indication of the one or more feature processing proposals to a client.

8. The method as recited in claim 7, wherein said implementing the feature processing plan is responsive to obtaining, from the client, an indication of approval of the particular feature processing proposal.

9. The method as recited in claim 6, further comprising performing, by the one or more computing devices:

receiving, via one or more programmatic interfaces of the machine learning service, a model creation request comprising respective indications of one or more of;

(a) the one or more target variables, (b) one or more prediction quality metrics including the particular prediction quality metric, (c) one or more prediction goals including the particular prediction goal, or (d) one or more constraints including a particular constraint identifying a mandatory feature processing transformation.

10. The method as recited in claim 9, wherein further comprising performing, by the one or more computing devices:

in response to determining that one or more feature processing proposals based at least in part on the model creation request are unacceptable to a client of the machine learning service,

transmitting a requirement reconsideration request to the client; and

receiving an indication from the client of a relative priority assigned to one or more of;

(a) the particular prediction quality metric, or (b) the particular prediction goal.

11. The method as recited in claim 6, wherein the particular prediction quality metric comprises one or more of:

(a) an AUC (area under curve) metric, (b) an accuracy metric, (c) a recall metric, (d) a sensitivity metric, (e) a true positive rate, (f) a specificity metric, (g) a true negative rate, (h) a precision metric, (i) a false positive rate, (j) a false negative rate, (k) an F1 score, (l) a coverage metric, (m) an absolute percentage error metric, or (n) a squared error metric.

12. The method as recited in claim 6, wherein the particular feature processing transformation comprises a use of one or more of:

(a) a quantile bin function, (b) a Cartesian product function, (c) a bi-gram function, (d) an n-gram function, (e) an orthogonal sparse bigram function, (f) a calendar function, (g) an image processing function, (h) an audio processing function, (i) a bio-informatics processing function, or (j) a natural language processing function.

13. The method as recited in claim 6, wherein the particular prediction goal comprises one or more of:

(a) a model execution time goal, (b) a memory usage goal, (c) a processor usage goal, (d) a storage usage goal, (e) a network usage goal, or (f) a budget.

14. The method as recited in claim 6, further comprising performing, by the one or more computing devices:

providing a programmatic interface enabling a client of the machine learning service to determine an extent to which the particular prediction goal is met by a particular execution of the machine learning model.

15. The method as recited in claim 6, wherein said determining the quality estimate comprises implementing a plurality of evaluation runs of the machine learning model, including a first evaluation run in which a first set of values of the particular processed variable are provided as input to the machine learning model, and a second evaluation run in which a different set of values of the particular processed variable are provided as input to the machine learning model.

16. The method as recited in claim 6, wherein said determining the cost estimate comprises implementing respective evaluation runs of a first variant of the machine learning model and a second variant of the machine learning model, wherein the first variant is trained using a first set of input variables that includes the particular processed variable, and the second variant is trained using a second set of input variables that excludes the particular processed variable.

17. The method as recited in claim 6, further comprising performing, by the one or more computing devices:

receiving, from a client, an indication of a recipe indicating one or more feature processing transformations requested by the client on the input variables of the training data set, wherein the particular feature processing transformation is not included in the recipe; and

providing, to the client, a proposed modification to the recipe, wherein the proposed modification includes an indication of the particular feature processing transformation.

18. A non-transitory computer-accessible storage medium storing program instructions that when executed on one or more processors:

identify, at a machine learning service, a set of candidate input variables usable to train a machine learning model to predict one or more target variables, wherein the set of candidate input variables includes at least a particular processed variable resulting from a particular feature processing transformation applicable to one or more input variables of a training data set;

determine a cost estimate indicative of an effect, on a performance metric associated with a particular prediction goal, of implementing the particular feature processing transformation; and

implement, based at least in part on the cost estimate, a feature processing proposal that excludes the particular feature processing transformation.

19. The non-transitory computer-accessible storage medium as recited in claim 18, wherein the instructions when executed on the one or more processors:

determine a quality estimate indicative of an effect, on a particular prediction quality metric, of implementing the particular feature processing transformation;

wherein the feature processing proposal is implemented based at least in part on the quality estimate.

20. The non-transitory computer-accessible storage medium as recited in claim 18, wherein the machine learning model comprises one or more of:

(a) a classification model, (b) a regression model, (c) a natural language processing (NLP) model, or (d) a clustering model.

21. The non-transitory computer-accessible storage medium as recited in claim 18, wherein the particular feature processing transformation comprises a use of one or more of:

(a) a quantile bin function, (b) a Cartesian product function, (c) a bi-gram function, (d) an n-gram function, (e) an orthogonal sparse bigram function, (f) a calendar function, (g) an image processing function, (h) an audio processing function, (i) a bio-informatics processing function, or (j) a natural language processing function.