METHODS AND SYSTEMS FOR PREDICTIVE ENGINE EVALUATION AND REPLAY OF ENGINE PERFORMANCE

• US 20170124486A1
• Filed: 01/11/2017
• Published: 05/04/2017
• Est. Priority Date: 03/20/2015
• Status: Active Grant

• Current Assignee: Salesforce.com, Inc.
• Original Assignee: Salesforce.com, Inc.
• Inventors: Chan, Ka Hou, Chan, Simon, Szeto, Kit Pang, Yip, Yue Kwen Justin
• Granted Patent: US 10,192,172 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 16/2455 Query execution
• CPC Class Codes: G06F 16/248 Presentation of query results
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 5/022 Knowledge engineering; Knowledge acquisition
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks
• CPC Class Codes: G06Q 30/0204 Market segmentation

Abstract

In accordance with disclosed embodiments, there are provided systems, methods, and apparatuses for implementing predictive engine evaluation and replay of engine performance. An exemplary system may include, for example: means selecting a first set of one or more algorithms for a machine learning model; tuning a first group of predictive engine parameters for the machine learning model; training the machine learning model with one or more sources of data using the selected first set of one or more algorithms and the first group of tuned predictive engine parameters to generate a first predictive engine variant from the trained machine learning model; selecting a second set of one or more algorithms for a machine learning model which are different than the first set; tuning a second group of predictive engine parameters for the machine learning model which are different than the first group; training the machine learning model with the one or more sources of data using the selected second set of one or more algorithms and the second group of tuned predictive engine parameters to generate a second predictive engine variant from the trained machine learning model; performing multiple experiments using the first and second predictive engine variants; comparing results from the multiple experiments; and deploying either the first predictive engine variant or the second predictive engine variant based on the comparison of the results of the multiple experiments. Other related embodiments are disclosed.

Claims

1. A method comprising:

selecting a first set of one or more algorithms for a machine learning model;

tuning a first group of predictive engine parameters for the machine learning model;

training the machine learning model with one or more sources of data using the selected first set of one or more algorithms and the first group of tuned predictive engine parameters to generate a first predictive engine variant from the trained machine learning model;

selecting a second set of one or more algorithms for a machine learning model which are different than the first set;

tuning a second group of predictive engine parameters for the machine learning model which are different than the first group;

training the machine learning model with the one or more sources of data using the selected second set of one or more algorithms and the second group of tuned predictive engine parameters to generate a second predictive engine variant from the trained machine learning model;

performing multiple experiments using the first and second predictive engine variants;

comparing results from the multiple experiments; and

deploying either the first predictive engine variant or the second predictive engine variant based on the comparison of the results of the multiple experiments.

2. The method of claim 1, wherein the one or more sources of data are selected from the group comprising:

training data;

test data;

real data;

live data;

historical data; and

simulated data.

3. The method of claim 1, further comprising:

outputting predictive results from the deployed first or second predictive engine variant.

4. The method of claim 1, further comprising:

receiving a replay request; and

replaying performance of the deployed predictive engine variant.