SYSTEMS, METHODS, AND APPARATUSES FOR IMPLEMENTING MACHINE LEARNING MODEL TRAINING AND DEPLOYMENT WITH A ROLLBACK MECHANISM

• US 20170124487A1
• Filed: 01/16/2017
• Published: 05/04/2017
• Est. Priority Date: 03/20/2015
• Status: Active Grant

• Current Assignee: Salesforce.com, Inc.
• Original Assignee: Salesforce.com, Inc.
• Inventors: Szeto, Kit Pang, Chan, Simon
• Granted Patent: US 10,713,594 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 11/1448 Management of the data involved in backup or backup restore
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06Q 30/0204 Market segmentation

Abstract

In accordance with disclosed embodiments, there are provided systems, methods, and apparatuses for implementing machine learning model training and deployment with a rollback mechanism within a computing environment. For example, an exemplary machine learning platform includes means for receiving training data as input at the machine learning platform, in which the training data includes a multiple transactions, each of the transactions specifying a plurality of features upon which to make a prediction and a label representing a correct answer for the plurality of features according to each respective transaction; specifying a model to be trained by the machine learning platform using the training data, in which the model includes a plurality of algorithms and source code; generating a new predictive engine variant by training the model to algorithmically arrive upon the label representing the correct answer as provided with the training data based on the plurality of features for each of the multiple transactions; versioning the new predictive engine variant based at least on the time the new predictive engine variant was generated a version of the source code utilized within the model and the training data received as input; deploying the new predictive engine variant into a production environment to replace a prior version of the predictive engine variant; and rolling back the new predictive engine variant from the production environment to a specified version which is less than a version of the new predictive engine variant. Other related embodiments are disclosed.

Claims

1. A method performed by a machine learning platform having at least a processor and a memory therein, wherein the method comprises:

receiving training data as input at the machine learning platform, wherein the training data includes a multiple transactions, each of the transactions specifying a plurality of features upon which to make a prediction and a label representing a correct answer for the plurality of features according to each respective transaction;

specifying a model to be trained by the machine learning platform using the training data, wherein the model includes a plurality of algorithms and source code;

generating a new predictive engine variant by training the model to algorithmically arrive upon the label representing the correct answer as provided with the training data based on the plurality of features for each of the multiple transactions;

versioning the new predictive engine variant based at least on the time the new predictive engine variant was generated a version of the source code utilized within the model and the training data received as input;

deploying the new predictive engine variant into a production environment to replace a prior version of the predictive engine variant; and

rolling back the new predictive engine variant from the production environment to a specified version which is less than a version of the new predictive engine variant.

2. The method of claim 1, wherein rolling back the new predictive engine variant comprises rolling back to a specified version identifier.

3. The method of claim 1, wherein rolling back the new predictive engine variant comprises rolling back to a version of the predictive engine variant having been created by training the model utilizing training data corresponding to a specified date range.

4. The method of claim 1:

wherein deploying the new predictive engine variant into a production environment comprises exposing the new predictive engine variant at a query interface; and

wherein the method further comprises;

receiving a query directed toward the new predictive engine variant having been deployed into the production environment, wherein the query requests a prediction from the new predictive engine variant;

wherein the query specifies a plurality of features; and

returning from the new predictive engine variant in response to the query, a label representing a predicted correct answer based on the features specified with the query.

5. The method of claim 1, wherein deploying the new predictive engine variant into a production environment comprises rendering predictive results as output from the new predictive engine variant, wherein the predictive results yield labels as output from the new predictive engine variant based on a plurality of never before seen features provided as input to the new predictive engine variant.

6. The method of claim 1, wherein deploying the new predictive engine variant into a production environment comprises deploying a latest trained model into production;

and

wherein rolling back the new predictive engine variant from the production environment comprises downrevving the latest trained model by re-deploying a prior trained model from a source code repository linked to the machine learning platform based on a previous_model_id corresponding to the prior trained model.

7. The method of claim 1:

wherein the machine learning platform operates within a host organization which provides on-demand cloud computing services to a plurality of tenants; and

wherein receiving the training data comprises receiving the training data as input from one of the plurality of tenants of the host organization.

8. The method of claim 7, further comprising:

receiving an initiation command to generate the new predictive engine variant from the one of the plurality of tenants of the host organization having provided the training data as input to the machine learning platform; and

wherein versioning the new predictive engine variant is based further upon the one of the plurality of tenants of the host organization from whom the training data was received and from whom the initiation command was received.

9. The method of claim 1, wherein the multiple transactions within the training data include any one or more of:

simulated transactions;

past actual user transactions;

user records;

product records;

behavioral data;

user activities data;

events data;

images;

voice data; and

text.

10. The method of claim 1, wherein the source code comprises a source code repository linked to the machine learning platform or a unique source code version within the source code repository for use in training the model.

11. The method of claim 1, further comprising:

receiving a replay request; and

replaying performance of the deployed new predictive engine variant.

12. The method of claim 1, further comprising:

receiving as configuration input at the machine learning platform any one or more of;

one or more engine parameters for tuning the machine learning platform;

one or more algorithmic parameters for tuning algorithms utilized by the machine learning platform;

one or more hyperparameters specifying data sources for training the machine learning platform, algorithms utilized by the machine learning platform, business logic parameters utilized by the machine learning platform, and/or data inputs to individual algorithms utilized by the machine learning platform; and

one or more prediction engine level comparisons for use by the machine learning platform.

13. The method of claim 12, wherein the configuration input received by the machine learning platform includes the specification of the configuration to alter generated prediction engine variants according to developer specification or according to developer selectable templates with changeable default values or according to automatically generated default configuration parameters provided by the machine learning platform.

14. The method of claim 1, further comprising:

receiving as configuration input at the machine learning platform any one or more of;

one or more algorithmic tuning parameters;

additional selection of data sources for use as training data input;

additional algorithms; and

one or more business rules to be observed by the machine learning platform when training the prediction engine.

15. The method of claim 1, wherein the input comprises an alphanumerical string representing all or part of a paragraph of free form text input from a user and a classification indicating whether the alphanumerical string corresponds to a customer complaint, a customer suggestion, or a customer compliment.

16. The method of claim 1, wherein the prediction output from the machine learning platform specifies a set of multiple item IDs, a set of recommended item IDs, a set of product IDs indicated as being predictively similar to input product IDs, one or more numerical scores returned responsive to a query and/or non-numerical criteria returned as the prediction output responsive to a query.

17. Non-transitory computer readable storage media having instructions stored thereupon that, when executed by a processor and memory of a machine learning platform, the instructions cause the machine learning platform to perform operations including:

receiving training data as input at the machine learning platform, wherein the training data includes a multiple transactions, each of the transactions specifying a plurality of features upon which to make a prediction and a label representing a correct answer for the plurality of features according to each respective transaction;

specifying a model to be trained by the machine learning platform using the training data, wherein the model includes a plurality of algorithms and source code;

generating a new predictive engine variant by training the model to algorithmically arrive upon the label representing the correct answer as provided with the training data based on the plurality of features for each of the multiple transactions;

versioning the new predictive engine variant based at least on the time the new predictive engine variant was generated a version of the source code utilized within the model and the training data received as input;

deploying the new predictive engine variant into a production environment to replace a prior version of the predictive engine variant; and

rolling back the new predictive engine variant from the production environment to a specified version which is less than a version of the new predictive engine variant.

18. The non-transitory computer readable storage media of claim 17, wherein rolling back the new predictive engine variant comprises rolling back to a specified version identifier.

19. The non-transitory computer readable storage media of claim 17, wherein rolling back the new predictive engine variant comprises rolling back to a version of the predictive engine variant having been created by training the model utilizing training data corresponding to a specified date range.

20. The non-transitory computer readable storage media of claim 17:

wherein deploying the new predictive engine variant into a production environment comprises exposing the new predictive engine variant at a query interface; and

wherein the instructions, when executed by the processor and the memory, cause the machine learning platform to perform additional operations including;

receiving a query directed toward the new predictive engine variant having been deployed into the production environment, wherein the query requests a prediction from the new predictive engine variant;

wherein the query specifies a plurality of features; and

returning from the new predictive engine variant in response to the query, a label representing a predicted correct answer based on the features specified with the query.

21. A machine learning platform comprising:

a memory to store instructions;

a processor to execute instructions;

a receive interface to receive training data as input at the machine learning platform, wherein the training data includes a multiple transactions, each of the transactions specifying a plurality of features upon which to make a prediction and a label representing a correct answer for the plurality of features according to each respective transaction;

the receive interface to receive a specified model to be trained by the machine learning platform using the training data, wherein the model includes a plurality of algorithms and source code;

a predictive engine generator to generate a new predictive engine variant by training the model to algorithmically arrive upon the label representing the correct answer as provided with the training data based on the plurality of features for each of the multiple transactions;

a versioning system interfaced with the machine learning platform to version the new predictive engine variant based at least on the time the new predictive engine variant was generated a version of the source code utilized within the model and the training data received as input;

a deployment platform to deploy the new predictive engine variant into a production environment to replace a prior version of the predictive engine variant; and

the deployment platform to roll back the new predictive engine variant from the production environment to a specified version maintained by the versioning system which is less than a version of the new predictive engine variant.

22. The machine learning platform of claim 21, wherein the deployment platform to roll back the new predictive engine variant comprises the deployment platform to roll back to a specified version identifier.

23. The machine learning platform of claim 21, wherein the deployment platform to roll back the new predictive engine variant comprises the deployment platform to roll back the new predictive engine variant to version of the predictive engine variant created by training the model utilizing training data corresponding to a specified date range.

24. The machine learning platform of claim 21:

wherein the deployment platform to deploy the new predictive engine variant into a production environment comprises the deployment platform to expose the new predictive engine variant at a query interface; and

wherein the machine learning platform further comprises a request interface to receive a query directed toward the new predictive engine variant having been deployed into the production environment, wherein the query requests a prediction from the new predictive engine variant, the query specifying a plurality of features; and

wherein the machine learning platform further comprises a query interface to return from the new predictive engine variant in response to the query, a label representing a predicted correct answer based on the features specified with the query.

25. The machine learning platform of claim 21, further comprising a query interface to render predictive results as output from the new predictive engine variant, wherein the predictive results yield labels as output from the new predictive engine variant based on a plurality of never before seen features provided as input to the new predictive engine variant.