DYNAMIC MODEL DATA FACILITY AND AUTOMATED OPERATIONAL MODEL BUILDING AND USAGE

• US 20140180982A1
• Filed: 10/23/2013
• Published: 06/26/2014
• Est. Priority Date: 12/20/2012
• Status: Active Grant

• Current Assignee: Aha Analytics Software LLC
• Original Assignee: Aha! Software LLC (TTEC Holdings, Inc.)
• Inventors: Teflian, Mark Samuel, Murphy, Robert Craig, Bacon, Bruce Allen, Gallanis, Peter T.
• Granted Patent: US 10,896,374 B2
• Time in Patent Office: Days
• US Class Current: 706/14
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 5/022 Knowledge engineering; Knowledge acquisition
• CPC Class Codes: G06N 5/04 Inference or reasoning models

Abstract

A commercial process with a dependent variable can be associated with a set of independent variables. The commercial process can continuously provide data collection opportunities. An intervention is designed using a model to predict the dependent outcome. The actual outcome of the intervention can be determined within the window of utility for these data. One objective is to improve intervention outcomes with prediction. Purely random outcomes (no model prediction) and outcomes resulting from the intervention (model operations) are aggregated into separate files—a sequence of control model data files and a sequence of model data files of operational data. These model data files and control model data files are used to analyze model performance and to react automatically when identified conditions warrant.

Claims

1. A method, comprising:

building a model data file, the model data file including normalized data values from two or more data sources that have been joined and aggregated across one or more periods of time; and

dynamically updating the model data file based on an analysis of the model data file being used in at least one expert system.

2. The method of claim 1, wherein the normalized data values in the model data file are configured such that information for an entity is conveyed in a single row.

3. The method of claim 1, further comprising:

determining analytics meta-data for the model data file, the analytics meta-data identifying a process for constructing one or more values for a single row or observation in the model data file; and

building a model data file with the analytics meta-data.

4. The method of claim 3, wherein the analytics meta-data defines at least one of the following:

at least one of the two or more data sources;

a transformation to be performed on data obtained from at least one of the two or more data sources;

a relationship of a first data set from a first data source to a second data set from a second data source; and

at least one criterion for selecting data from at least one of the two or more data sources.

5. The method of claim 1, wherein the normalized data values are calculated based on raw data obtained from at least one of the two or more data sources.

6. The method of claim 1, wherein at least one of the two or more data sources correspond to at least one of a Customer Relations Management (CRM) database, Big Data, a billing database, a support database, a transactional database, a delivery database, and a quality database.

7. The method of claim 1, wherein data in a row of the model data file is summed and counted within a context.

8. The method of claim 7, wherein the context comprises at least one of new entries and entries within a predefined range of entries.

9. A method, comprising:

building a model;

using the model in a first expert system;

obtaining at least some ongoing operational data from the first expert system while the first expert system uses the model; and

repeatedly testing the model against actual data to determine whether or not to recalibrate or re-specify the model.

10. The method of claim 9, wherein a second expert system performs the testing of the model.

11. The method of claim 10, wherein the second expert system obtains random samples of data from the first expert system to test the model.

12. The method of claim 9, further comprising:

obtaining an independent source of random data; and

using the independent source of random data as part of the testing.

13. The method of claim 9, wherein the model is tested by tracking a goodness-of-fit (GOF).

14. The method of claim 13, further comprising:

gathering one or more statistics relevant to calibration for the actual data;

gathering one or more statistics relevant to discrimination for the actual data;

comparing the one or more statistics relevant to calibration to a first pre-existing validation data;

comparing the one or more statistics relevant to discrimination to a second pre-existing validation data; and

based on the comparisons, determining the GOF for the model.

15. An operational intelligence platform, comprising:

a first expert system configured to use a model in connection with providing analytics data to one or more interested parties; and

a second expert system configured to periodically test and validate the model based on changes in one or more independent variables and their relation to one or more dependent variables.

16. The platform of claim 15, wherein the second expert system is configured to obtain an independent source of random data to account for a modeler effect during the testing and validation of the model.

17. The platform of claim 15, wherein the second expert system is configured to test and validate the model at least on a daily basis.