MACHINE LEARNING SYSTEM INTERFACE

• US 20160358101A1
• Filed: 06/05/2015
• Published: 12/08/2016
• Est. Priority Date: 06/05/2015
• Status: Active Grant

• Current Assignee: Meta Platforms, Inc. (f/k/a Facebook, Inc.)
• Original Assignee: Meta Platforms, Inc. (f/k/a Facebook, Inc.)
• Inventors: Xie, Xiaowen, Bowers, Stuart Michael, Mehanna, Hussein Mohamed Hassan, Azzolini, Alisson Gusatti, Dunn, Jeffrey Scott, Farnham, Rodrigo Bouchardet, Paton, James Robert, Sidorov, Aleksandr, Vagata, Pamela Shen
• Granted Patent: US 10,417,577 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 8/34 Graphical or visual programming
• CPC Class Codes: G06F 9/453 Help systems
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 3/12 using genetic models

Abstract

Some embodiments include an experiment management interface for a machine learning system. The experiment management interface can manage one or more workflow runs related to building or testing machine learning models. The experiment management interface can receive an experiment initialization command to create a new experiment associated with a new workflow. A workflow can be represented by an interdependency graph of one or more data processing operators. The experiment management interface enables definition of the new workflow from scratch or by cloning and modifying an existing workflow. The workflow can define a summary format for its inputs and outputs. In some embodiments, the experiment management interface can automatically generate a comparative visualization at the conclusion of running the new workflow based on an input schema or an output schema of the new workflow.

Claims

1. A computer-implemented method, comprising:

generating an experiment management interface to present previous workflows ran on a machine learning system;

receiving, via the experiment management interface and with the machine learning system, an experiment initialization command to create a new experiment associated with a new workflow, wherein the experiment initialization command includes a selection of an existing workflow in the machine learning system, wherein the existing workflow is represented by an interdependency graph of one or more data processing operators;

receiving one or more modifications to the existing workflow via the experiment management interface;

scheduling the new workflow based on the modifications to the existing workflow for execution by a dynamic pool of computing devices; and

generating a visualization to facilitate analysis of the new experiment based on an input schema or an output schema of the new workflow.

2. The computer-implemented method of claim 1, wherein generating the experiment management interface includes exposing the experiment management interface usable to identify previously executed experiments on the machine learning system;

and wherein receiving the experiment initialization command includes receiving a user selection of a previous experiment associated with the existing workflow.

3. The computer-implemented method of claim 1, wherein said receiving the experiment initialization command includes receiving a user selection of the existing workflow at the experiment management interface;

and wherein said receiving the modifications includes, responsive to receiving the user selection, presenting a combination of at least a subset of parameters and attributes of the existing workflow for cloning or modification.

4. The computer-implemented method of claim 1, wherein the modifications includes redefining an existing data processing operator used in the existing workflow.

5. The computer-implemented method of claim 1, wherein the modifications includes changing an input dataset to the existing workflow.

6. The computer-implemented method of claim 1, wherein the modifications includes changing an input data usage parameter to the existing workflow.

7. The computer-implemented method of claim 1, wherein the modifications includes removing an existing data processing operator used in the existing workflow.

8. The computer-implemented method of claim 1, wherein the modifications includes inserting an additional data processing operator into the existing workflow, wherein the modifications includes definition of the additional data processing operator.

9. The computer-implemented method of claim 1, wherein the modifications includes inserting an additional data processing operator into the existing workflow, wherein the modifications includes a reference to a production data processing operator from a production operator library as the additional data processing operator.

10. The computer-implemented method of claim 1, further comprising rendering an integrated development environment (IDE) text editor in the experiment management interface to enable modification to the interdependency graph of the data processing operators;

and wherein the modifications is received via the IDE text editor.

11. The computer-implemented method of claim 1, further comprising tracking a version history of the new workflow, wherein the version history identifies the existing workflow as a basis for the new workflow.

12. The computer-implemented method of claim 1, wherein generating the visualization schema includes:

identifying one or more comparable experiments based on output schemas of one or more experiments tracked in an experiment repository in the machine learning system, wherein the output schemas of the comparable experiments match an output schema of the new experiment; and

responsive to a user selection of a target comparable experiment, rendering a visualization to compare the new experiment and the target comparable experiment, wherein said rendering is based on a summary format dictated by the new workflow associated with the new experiment.

13. The computer-implemented method of claim 1, wherein generating the visualization schema includes:

identifying one or more comparable experiments for user selection based on a version history of the new experiment, wherein the version history tracks a provenance chain of one or more experiments that are based on one another;

receiving a user selection of a target comparable experiment from amongst the comparable experiments; and

responsive to receiving the user selection, rendering a visualization to compare the new experiment and the target comparable experiment.

14. A computer readable data memory storing computer-executable instructions that, when executed, cause a computer system to perform a computer-implemented method, the instructions comprising:

instructions for generating an experiment management interface to facilitate machine learning experimentations in a machine learning system;

instructions for registering an experiment with an experiment management engine of the machine learning system;

instructions for receiving, via an integrated development environment of the experiment management interface, a workflow definition text that defines a new workflow, wherein the workflow definition text describes an interdependency graph of one or more data processing operators;

instructions for compiling the new workflow for execution on a dynamic pool of computing devices; and

instructions for generating a visualization schema to facilitate analysis of the experiment based on an input schema or an output schema of the new workflow.

15. The computer readable data memory of claim 14, wherein the workflow definition is received as text.

16. The computer readable data memory of claim 14, wherein the workflow definition is received as one or more selections of an input dataset, a data processing parameter, one or more data processing operators, an experiment report configuration, or any combination thereof, that is known in a workflow repository of the machine learning system.

17. The computer readable data memory of claim 14, further comprising:

instructions for storing the experiment in an experiment repository of the machine learning system; and

instructions for presenting the experiment, after the new workflow of the experiment has executed, at the experiment management interface to enable another experimenter to repeat the experiment or repeat a modification of the experiment for automatic comparison.

18. A computer system, comprising:

an experiment repository configured to store machine learning experiments previously executed on a machine learning system;

an experiment management engine configured to generate an experiment definition interface for defining a new experiment and at least an associated workflow and an experiment visualization interface for analyzing a result of the new experiment;

an experiment scheduler engine configured to schedule execution of the experiment via a pool of computing devices and distributing one or more code packages to at least a subset of the pool of computing devices; and

wherein the experiment definition interface enables an experimenter to register the new experiment with a machine learning system, wherein the new experiment is based on one of the previous machine learning experiments, wherein the previous machine learning experiment is associated with an existing workflow represented by an interdependency graph of one or more data processing operators; and

wherein the experiment visualization interface generates a visualization image comparing a first resulting output of a comparable previous experiment and a second resulting output of running the new experiment according to a visualization schema based on an input schema or an output schema of the associated workflow.

19. The computer system of claim 18, wherein the experiment definition interface includes a search widget capable of identifying one or more workflows based on a user query of an input dataset, an input data source, an input schema, an output schema, an input summary format, an output summary format, or any combination thereof, that match one or more respective parameters of the identified workflows.

20. The computer system of claim 18, wherein the experiment definition interface includes a search widget capable of identifying one or more data processing operators based on a user query of an input dataset, an input data source, an input schema, an output schema, an input summary format, an output summary format, or any combination thereof, that match one or more respective parameters of the identified data processing operators.