Interactive interfaces for machine learning model evaluations

• US 10,452,992 B2
• Filed: 11/11/2014
• Issued: 10/22/2019
• Est. Priority Date: 06/30/2014
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Lee, Polly Po Yee, Correa, Nicolle M., Dirac, Leo Parker, Ingerman, Aleksandr Mikhaylovich
• Primary Examiner(s): Starks, Wilbert L
• Application Number:
  US14/538,723
• Publication Number:
  US 20150379429A1
• Time in Patent Office: 1,806 Days
• Field of Search: None
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 5/04 Inference or reasoning models
• CPC Class Codes: G09B 5/00 Electrically-operated educational appliances working with questions and answers G09B7/00; simulators G09B9/00; advertising or displaying in general G09F

Abstract

A first data set corresponding to an evaluation run of a model is generated at a machine learning service for display via an interactive interface. The data set includes a prediction quality metric. A target value of an interpretation threshold associated with the model is determined based on a detection of a particular client'"'"'s interaction with the interface. An indication of a change to the prediction quality metric that results from the selection of the target value may be initiated.

Claims

1. A system, comprising:

one or more computing devices configured to;

train a machine learning model to generate values of one or more output variables corresponding to respective observation records at a machine learning service of a provider network, wherein the one or more output variables include a particular output variable;

subsequent to the training;

generate an interactive graphical interface that displays a first set of data produced by one or more evaluation runs of the machine learning model performed using respective evaluation data sets, wherein the first set of data comprises at least (a) a statistical distribution of the particular output variable, and (b) a first prediction quality metric of the machine learning model;

receive input from a client via a first graphical control of the interactive graphical interface to modify a first prediction interpretation threshold, wherein the first prediction interpretation threshold specifies how the particular output variable is interpreted, wherein the modification is performed without modifying the machine learning model;

determine, based at least in part on the input from the client via the first graphical control, a target value of the first prediction interpretation threshold;

initiate a display, via the interactive graphical interface, of a change to the first prediction quality metric resulting from a selection of the target value;

in response to a request transmitted by the client via the interactive graphical interface, save the target value in a persistent repository of the machine learning service; and

utilize the saved target value to interpret the particular output variable and generate one or more prediction results of a subsequent run of the machine learning model.

2. The system as recited in claim 1, wherein the machine learning model is a binary classification model that is to be used to classify observation records into a first category and a second category, and wherein the first prediction interpretation threshold indicates a cutoff boundary between the first and second categories.

3. The system as recited in claim 1, wherein the first prediction quality metric comprises one or more of:

an accuracy metric, a recall metric, a sensitivity metric, a true positive rate, a specificity metric, a true negative rate, a precision metric, a false positive rate, a false negative rate, an F1 score, a coverage metric, an absolute percentage error metric, a squared error metric, or an AUC (area under a curve) metric.

4. The system as recited in claim 1, wherein the first graphical control comprises a continuous-variation control element enabling the client to indicate a transition between a first value of the first prediction interpretation threshold and a second value of the first prediction interpretation threshold, wherein the one or more computing devices are further configured to:

initiate an update, in real time, as the client indicates a transition from the first value to the second value, of a portion of the interactive graphical interface indicating a corresponding change to the first prediction quality metric.

5. The system as recited in claim 1, wherein the interactive graphical interface comprises respective additional controls for indicating target values of a plurality of prediction quality metrics including the first prediction quality metric and a second prediction quality metric, wherein the one or more computing devices are further configured to:

in response to a change, indicated using a first additional control, of a target value of the first prediction quality metric, initiate an update of a display of a second additional control corresponding to the second prediction quality metric, indicating an impact of the change of the target value of the first prediction quality metric on the second prediction quality metric.

6. A method, comprising:

performing, by one or more computing devices;

training a machine learning model to generate values of one or more output variables corresponding to respective observation records, wherein the one or more output variables include a particular output variable;

subsequent to the training;

generating an interactive graphical interface that displays a first set of data produced by one or more evaluation runs of the machine learning model, wherein the first set of data includes at least a first prediction quality metric of the machine learning model;

receiving input from a client via a first graphical control of the interactive graphical interface to modify a first prediction interpretation threshold, wherein the first prediction interpretation threshold specifies how the particular output variable is interpreted, wherein the modification is performed without modifying the machine learning model;

determining, based at least in part on the input from the client via the first graphical control, a target value of the first prediction interpretation threshold;

initiating a display, via the interactive graphical interface, of a change to the first prediction quality metric resulting from a selection of the target value; and

obtaining, using the target value, interpretations of the particular output variable to generate one or more prediction results of a subsequent run of the machine learning model.

7. The method as recited in claim 6, wherein the machine learning model is a binary classification model that is to be used to classify observation records into a first category and a second category, and wherein the first prediction interpretation threshold indicates a cutoff boundary between the first and second categories.

8. The method as recited in claim 6, wherein the first prediction quality metric comprises one or more of:

an accuracy metric, a recall metric, a sensitivity metric, a true positive rate, a specificity metric, a true negative rate, a precision metric, a false positive rate, a false negative rate, an F1 score, a coverage metric, an absolute percentage error metric, a squared error metric, or an AUC (area under a curve) metric.

9. The method as recited in claim 6, wherein the first graphical control comprises a continuous-variation control element enabling the client to indicate a transition between a first value of the first prediction interpretation threshold and a second value of the first prediction interpretation threshold, further comprising performing, by the one or more computing devices:

initiating an update, in real time, as the client indicates a transition from the first value to the second value, of a portion of the interactive graphical interface indicating a corresponding change to the first prediction quality metric.

10. The method as recited in claim 6, wherein the interactive graphical interface comprises respective additional controls for indicating target values of a plurality of prediction quality metrics including the first prediction quality metric and a second prediction quality metric, further comprising performing, by the one or more computing devices:

in response to a change, indicated using a first additional control, of a target value of the first prediction quality metric, initiating an update of a display of a second additional control corresponding to the second prediction quality metric, indicating an impact of the change of the target value of the first prediction quality metric on the second prediction quality metric.

11. The method as recited in claim 10, further comprising performing, by the one or more computing devices:

in response to the change, indicated using a first additional control, of the target value of the first prediction quality metric, initiating a display of a change of the first prediction interpretation threshold.

12. The method as recited in claim 6, wherein the machine learning model is one of:

(a) an n-way classification model or (b) a regression model.

13. The method as recited in claim 6, wherein the interactive graphical interface includes a region displaying a statistical distribution of values of the particular output variable, further comprising performing, by the one or more computing devices:

initiating a display, in response to a particular client interaction with the region, wherein the particular client interaction indicates a first value of the particular output variable, of values of one or more input variables of an observation record for which the particular output variable has the first value.

14. The method as recited in claim 6, further comprising performing, by the one or more computing devices:

generating, for display via the interactive graphical interface, an alert message indicating an anomaly detected during an execution of the machine learning model.

15. The method as recited in claim 6, further comprising performing, by the one or more computing devices:

receiving, in response to a use of a different control of the interactive graphical interface by the client subsequent to a display of the first prediction quality metric, a request to perform one or more of;

(a) a re-evaluation of the machine learning model or (b) a re-training of the machine learning model.

16. The method as recited in claim 6, further comprising performing, by the one or more computing devices:

saving, in a repository of a machine learning service implemented at a provider network, a record indicating the target value.

17. A non-transitory computer-accessible storage medium storing program instructions that when executed on one or more processors:

generate an interactive graphical interface that displays a first set of data produced by an evaluation run of a machine learning model subsequent to a training of the machine learning model, wherein the first set of data includes at least a first prediction quality metric of the machine learning mode;

receive input from a client via a first graphical control of interactive graphical interface to modify a first interpretation threshold, wherein the first prediction interpretation threshold specifies how an output variable generated by the machine learning model is interpreted to generate prediction results, wherein the modification is performed without modifying the machine learning model;

determine, based on the input from the client via the first graphical control, a target value of the first interpretation threshold; and

initiate a display, via the interactive graphical interface, of a change to the first prediction quality metric resulting from a selection of the target value.

18. The non-transitory computer-accessible storage medium as recited in claim 17, wherein the machine learning model is a binary classification model that is to be used to classify observation records into a first category and a second category, and wherein the first interpretation threshold indicates a cutoff boundary between the first and second categories.

19. The non-transitory computer-accessible storage medium as recited in claim 17, wherein the first prediction quality metric comprises one or more of:

an accuracy metric, a recall metric, a sensitivity metric, a true positive rate, a specificity metric, a true negative rate, a precision metric, a false positive rate, a false negative rate, an F1 score, a coverage metric, an absolute percentage error metric, a squared error metric, or an AUC (area under a curve) metric.

20. The non-transitory computer-accessible storage medium as recited in claim 17, wherein the first graphical control comprises a continuous-variation control element enabling the client to indicate a transition between a first value of the first interpretation threshold and a second value of the first interpretation threshold, wherein the instructions when executed on one or more processors:

initiate an update, in real time, as the particular user indicates a transition from the first value to the second value, of a portion of the interactive graphical interface indicating a corresponding change to the first prediction quality metric.

21. A non-transitory computer-accessible storage medium storing program instructions that when executed on one or more processors:

display a first set of data via an interactive interface during a particular interaction session with a client, wherein the first set of data corresponds to an evaluation run of a machine learning model subsequent to a training of the machine learning model, wherein the first set of data includes at least a first prediction quality metric associated with the evaluation run;

transmit, to a server of a machine learning service during the particular interaction session, based on a detection of a particular interaction of the client with the interactive interface, a target value of a first interpretation threshold, wherein the first prediction interpretation threshold specifies how an output variable generated by the machine learning model is interpreted to generate prediction results, wherein the modification is performed without modifying the machine learning model;

receive, from the server, an indication of a change to the first prediction quality metric resulting from a selection of the target value; and

indicate, via the interactive interface, the change to the first prediction quality metric during the particular interaction session.

22. The non-transitory computer-accessible storage medium as recited in claim 21, wherein the interactive interface comprises a graphical interface, and wherein the particular interaction comprises a manipulation of a first graphical control included in the graphical interface.

23. The non-transitory computer-accessible storage medium as recited in claim 21, wherein the interactive interface comprises a command-line interface.

24. The non-transitory computer-accessible storage medium as recited in claim 21, wherein the interactive interface comprises an API (application programming interface).