MACHINE LEARNING MODEL TRACKING PLATFORM

• US 20160300156A1
• Filed: 04/10/2015
• Published: 10/13/2016
• Est. Priority Date: 04/10/2015
• Status: Active Grant

• Current Assignee: Meta Platforms, Inc. (f/k/a Facebook, Inc.)
• Original Assignee: Meta Platforms, Inc. (f/k/a Facebook, Inc.)
• Inventors: Agarwal, Parul, Oberoi, Parv Ajay, Bowers, Stuart Michael, Mehanna, Hussein Mohamed Hassan
• Granted Patent: US 9,996,804 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/2465 Query processing support for facilitating data mining operations in structured databases
• CPC Class Codes: G06F 8/00 Arrangements for software engineering testing or debugging G06F11/36; administrative, planning or organisation aspects of software project management G06Q10/06
• CPC Class Codes: G06F 9/46 Multiprogramming arrangements
• CPC Class Codes: G06N 20/00 Machine learning

Abstract

Some embodiments include a machine learner platform. The machine learner platform can implement a model tracking service to track one or more machine learning models for one or more application services. A model tracker database can record a version history and/or training configurations of the machine learning models. The machine learner platform can implement a platform interface configured to present interactive controls for building, modifying, evaluating, deploying, or compare the machine learning models. A model trainer engine can task out a model training task to one or more computing devices. A model evaluation engine can compute an evaluative metric for a resulting model from the model training task.

Claims

1. A computer-implemented method, comprising:

receiving a training configuration based on modifying a production copy template of a production machine learning model for an application service;

scheduling a recurring training session based on the training configuration to produce a latent model;

tracking one or more differences in training configurations of the latent model as compared to the production copy template;

computing an evaluative metric of the latent model by performing an offline testing of the latent model as compared to production copy; and

generating a machine learner interface to access a model tracker database that indexes the evaluative metric and the tracked differences associated with the latent model, wherein the machine learner interface provides an interface element to trigger launching the latent model into production and cloning the latent model to replace the production copy template.

2. The computer-implemented method of claim 1, further comprising:

tracking a version history of the latent model, wherein the version history is represented by a provenance chain of one or more machine learning models that are based on one another in order; and

wherein tracking the version history includes tracking one or more modifications from a previous machine learning model in the provenance chain to a subsequent machine learning model in the provenance chain.

3. The computer-implemented method of claim 2, further comprising:

identifying the latent model as being corrupt; and

identifying a problem data source or a problem data feature based on a tracked training configuration modification of the latent model as compared to a previously working model in the version history.

4. The computer-implemented method of claim 3, further comprising:

in response to identifying the latent model as being corrupt, rolling back the latent model from production; and

pushing the previously working model into production for the application service.

5. The computer-implemented method of claim 1, further comprising detecting corruption of the latent model based on the evaluative metric of the latent model.

6. The computer-implemented method of claim 1, wherein the machine learner interface enables a developer or analyst user of the application service to build, evaluate, and deploy the latent model.

7. The computer-implemented method of claim 1, further comprising generating a comparison report by comparing at least one of evaluative metrics or training configurations between the production copy template and the latent model.

8. The computer-implemented method of claim 1, further comprising:

receiving, via the machine learner interface, a model selection of a target model other than the production copy and the latent model; and

rendering a comparison report between the latent model and the target model.

9. The computer-implemented method of claim 1, wherein the tracked differences in the training configurations of the latent model and the production copy includes differences in one or more sources of training datasets, one or more training datasets, or one or more data features, that were used to train the latent model.

10. A machine learner platform system, comprising:

a model tracking engine configured to track one or more machine learning models for one or more application services;

a platform interface configured to present interactive controls for building, modifying, evaluating, or deploying the machine learning models;

a model tracker database configured to record version history of the machine learning models tracked by the model tracking engine;

a model trainer engine configured to task out a model training task to one or more computing devices, wherein the model tracking engine is configured to track a training configuration of a resulting model from the model training task in the model tracker database; and

a model evaluation engine configured to compute an evaluative metric for the resulting model, wherein the platform interface presents the evaluative metric for the resulting model with the training configuration to facilitate evaluation of the resulting model.

11. The machine learner platform system of claim 10, wherein the platform interface is configured to present a comparison report of two or more of the machine learning models;

and wherein the model evaluation engine is configured to generate the comparison report by comparing results of running a test dataset through the two or more of the machine learning models.

12. The machine learner platform system of claim 10, wherein the model training task is a recurring task.

13. The machine learner platform system of claim 10, wherein the platform interface is configured to receive an indication that a dataset is defective;

and wherein, in response to the platform interface receiving the indication, the model tracking engine is configured to query the model tracker database to identify one or more of the machine learning models that used the dataset to train and to mark the identified one or more of the machine learning models as potentially defective in the platform interface.

14. The machine learner platform system of claim 10, wherein the platform interface is configured to receive a provenance query targeting a target machine learning model;

and wherein the platform interface is configured to render a diagram representing the version history from the model tracker database, the diagram illustrating one or more related machine learning models of the target machine learning model.

15. The machine learner platform system of claim 10, wherein the platform interface is configured to:

receive a search query targeting a particular dataset or a particular data feature; and

in response to receiving the search query, present one or more target machine learning models and evaluation statistics of the target machine learning models by querying the model tracker database to identify the target machine learning models that utilized the particular dataset or the particular data feature.

16. The machine learner platform system of claim 15, wherein the platform interface is further configured to present one or more deployment statuses corresponding to the target machine learning models in response to receiving the search query.

17. The machine learner platform system of claim 10, wherein the model evaluation engine is configured to compute a ranking of at least a subset of the machine learning models based on evaluative metrics corresponding to the subset of the machine learning models stored in the model tracker database;

and wherein the platform interface is configured to present the ranking.

18. A computer readable data storage memory storing computer-executable instructions that, when executed, cause a computer system to perform a computer-implemented method, the instructions comprising:

instructions for identifying a list of models tracked by a machine learner platform servicing one or more application services;

instructions for routing live traffic to one or more of the models according to one or more live testing designations or production designations of the one or more of the models;

instructions for determining that a target model of the models has not being used to serve live traffic within a threshold period of time; and

instructions for sending a notification to a project owner of the target model, wherein the notification includes a link to terminate resource consumption corresponding to maintenance of the target model.

19. The computer readable data storage memory of claim 18, wherein the instructions further comprises instructions for classifying the models based on whether a model is in production, is undergoing live testing, is undergoing recurring training, is idling, or any combination thereof.

20. The computer readable data storage memory of claim 18, wherein the instructions further comprises instructions for launching a live testing or production deployment of one of the models, in response to a user command received through a platform interface.