Autonomous decision support system using configuration inflation based ETL and content modeling

• US 9,910,903 B1
• Filed: 03/02/2015
• Issued: 03/06/2018
• Est. Priority Date: 03/02/2015
• Status: Active Grant

• Current Assignee: CA, Inc. (d/b/a CA Technologies) (Broadcom, Inc.)
• Original Assignee: CA, Inc. (d/b/a CA Technologies) (Broadcom, Inc.)
• Inventors: Jilani, Abdul Khader, Palaniappan, MuthuVadugan
• Primary Examiner(s): Pham, Khanh
• Application Number:
  US14/635,099
• Time in Patent Office: 1,100 Days
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses

Abstract

A computer-implemented method is disclosed which detects a change in a set of data sources of a DSS, such as addition of a new data source, removal of an existing data source, or a schema change of an existing data source in the set. A first set of DSS updates for pending ETL jobs of an ETL engine of the DSS are automatically determined based on the change in the set of data sources. The method automatically updates, without canceling, the pending ETL jobs of the ETL engine based on the first set of DSS updates. A second set of DSS updates for a semantic layer of a BI engine of the DSS are automatically determined based on the change in the set of data sources and based on the first set of DSS updates. The semantic layer is automatically updated based on the second set of DSS updates.

Claims

1. A computer-implemented method comprising:

detecting a change in a set of data sources of a decision support system (DSS), the change comprising addition of a new data source to the set, removal of an existing data source from the set, or a schema change of an existing data source in the set;

automatically determining a first set of DSS updates for pending extract, transform, load (ETL) jobs of an ETL engine of the DSS based on the change in the set of data sources, wherein the pending ETL jobs import data from the set of data sources into a data warehouse, and wherein automatically determining the first set of DSS updates for pending ETL lobs of the ETL engine comprises;

analyzing historical DSS updates performed for previous changes to the set of data sources;

attempting to determine first potential DSS updates using a primary artificial intelligence (AI) model; and

if the attempting identifies first potential DSS updates that have a confidence metric that is above a predefined threshold, including the first potential DSS updates in the first set of DSS updates;

automatically updating, without canceling, the pending ETL jobs of the ETL engine based on the first set of DSS updates;

automatically determining a second set of DSS updates for a semantic layer of a business intelligence (BI) engine of the DSS based on the change in the set of data sources and based on the first set of DSS updates, wherein the semantic layer comprises a mapping of fields from the data warehouse to reporting fields of the BI engine; and

automatically updating the semantic layer of the BI engine based on the second set of DSS updates.

2. The computer-implemented method of claim 1, wherein automatically updating, without canceling, the pending ETL jobs of the ETL engine based on the first set of DSS updates comprises automatically transmitting DSS reconfiguration instructions to the ETL engine that instruct the ETL engine to update the pending ETL jobs without cancelation of the pending ETL jobs.

3. The computer-implemented method of claim 1, wherein the ETL engine is configured to execute the pending ETL jobs in a configuration inflation mode of operation.

4. The computer-implemented method of claim 1, further comprising:

automatically transmitting schema update instructions to the ETL engine based on the first set of DSS updates, wherein the schema update instructions instruct the ETL engine to update a schema of the data warehouse to reflect the change in the set of data sources.

5. The computer-implemented method of claim 1, wherein automatically updating the semantic layer of the BI engine based on the second set of DSS updates comprises:

automatically invoking an Application Programming Interface (API) of the BI engine to cause the BI engine to update its semantic layer.

6. The computer-implemented method of claim 1, wherein automatically updating the semantic layer of the BI engine yields an updated semantic layer, the method further comprising:

automatically invoking an Application Programming Interface (API) of the BI engine to cause the BI engine to update, based on the updated semantic layer, its existing report templates that comprise data from the data warehouse.

7. The computer-implemented method of claim 1, wherein automatically determining the first set of DSS updates further comprises:

if the attempting to determine first potential DSS updates fails, or if first potential DSS updates are determined but a confidence metric of the first potential DSS updates is below a predefined threshold, using a secondary AI model to determine second potential DSS updates, and including the second potential DSS updates in the first set of DSS updates.

8. The computer-implemented method of claim 1, wherein the data source that is added, removed, or has its schema changed is a table, and wherein automatically determining the first set of DSS updates based on the change in the set of data sources comprises:

determining a DSS update rule for the change in the set of data sources as a function of a data model of the data warehouse, a table type of the table, and the change that was detected in the table; and

determining the first set of DSS updates based on the DSS update rule.

9. The computer-implemented method of claim 1, wherein detecting the change in the set of data sources comprises:

analyzing metadata of the set of data sources of the DSS; and

detecting the change in the set of data sources based on a change in the metadata of the set of data sources.

10. A computing device comprising:

a communication interface configured to communicate with an extract, transform, load (ETL) engine of a decision support system (DSS), a business intelligence (BI) engine of the DSS, or both the ETL engine and the BI engine; and

a processing circuit communicatively connected to the communication interface and configured to;

detect a change in a set of data sources of the DSS, the change comprising addition of a new data source to the set, removal of an existing data source from the set, or a schema change of an existing data source in the set;

automatically determine a first set of DSS updates for pending ETL jobs of the ETL engine based on the change in the set of data sources, wherein the pending ETL jobs import data from the set of data sources into a data warehouse, and wherein to automatically determine the first set of DSS updates for pending ETL jobs of the ETL engine, the processing circuit is configured to;

analyze historical DSS updates performed for previous changes to the set of data sources;

attempt to determine first potential DSS updates using a primary artificial intelligence (AI) model; and

if the attempt identifies first potential DSS updates that have a confidence metric that is above a predefined threshold, include the first potential DSS updates in the first set of DSS updates;

automatically update, without canceling, the pending ETL jobs of the ETL engine based on the first set of DSS updates;

automatically determine a second set of DSS updates for a semantic layer of the BI engine based on the change in the set of data sources and based on the first set of DSS updates, wherein the semantic layer comprises a mapping of fields from the data warehouse to reporting fields of the BI engine; and

automatically update the semantic layer of the BI engine based on the second set of DSS updates.

11. The computing device of claim 10, wherein to automatically update, without canceling, the pending ETL jobs of the ETL engine based on the first set of DSS updates, the processing circuit is configured to automatically transmit DSS reconfiguration instructions to the ETL engine that instruct the ETL engine to update the pending ETL jobs without cancelation of the pending ETL jobs.

12. The computing device of claim 10, wherein the ETL engine is configured to execute the pending ETL jobs in a configuration inflation mode of operation.

13. The computing device of claim 10, wherein the processing circuit is further configured to:

automatically transmit schema update instructions to the ETL engine based on the first set of DSS updates, wherein the schema update instructions instruct the ETL engine to update a schema of the data warehouse to reflect the change in the set of data sources.

14. The computing device of claim 10, wherein to automatically update the semantic layer of the BI engine based on the second set of DSS updates, the processing circuit is configured to:

automatically invoke an Application Programming Interface (API) of the business engine to cause the business engine to update its semantic layer.

15. The computing device of claim 10, wherein the automatic update of the semantic layer of the BI engine yields an updated semantic layer, and wherein the processing circuit is further configured to:

automatically invoke an Application Programming Interface (API) of the BI engine to cause the BI engine to update, based on the updated semantic layer, its existing report templates that comprise data from the data warehouse.

16. The computing device of claim 10, wherein to automatically determine the first set of DSS updates, the processing circuit is further configured to:

if the attempt to determine first potential DSS updates fails, or if first potential DSS updates are determined but a confidence metric of the first potential DSS updates is below a predefined threshold, use a secondary AI model to determine second potential DSS updates, and include the second potential DSS updates in the first set of DSS updates.

17. The computing device of claim 10, wherein the data source that is added, removed, or has its schema changed is a table, and wherein to automatically determine the first set of DSS updates based on the change in the set of data sources, the processing circuit is configured to:

determine a DSS update rule for the change in the set of data sources as a function of a data model of the data warehouse, a table type of the table, and the change that was detected in the table; and

determine the first set of DSS updates based on the DSS update rule.

18. The computing device of claim 10, wherein to detect the change in the set of data sources, the processing circuit is configured to:

analyze metadata of the set of data sources of the DSS; and

detect the change in the set of data sources based on a change in the metadata of the set of data sources.

19. A computer program product comprising:

a computer readable storage medium having computer readable program code embodied therewith that, when executed on processing circuitry, the processing circuitry to;

detect a change in a set of data sources of a decision support system (DSS), the change comprising addition of a new data source to the set, removal of an existing data source from the set, or a schema change of an existing data source in the set;

automatically determine a first set of DSS updates for pending extract, transform, load (ETL) jobs of an ETL engine of the DSS based on the change in the set of data sources, wherein the pending ETL jobs import data from the set of data sources into a data warehouse, and wherein to automatically determine the first set of DSS updates for pending ETL jobs of the ETL engine, the computer readable program code causes the processing circuitry to;

analyze historical DSS updates performed for previous changes to the set of data sources;

attempt to determine first potential DSS updates using a primary artificial intelligence (AI) model; and

if the attempt identifies first potential DSS updates that have a confidence metric that is above a predefined threshold, include the first potential DSS updates in the first set of DSS updates;

automatically update, without canceling, the pending ETL jobs of the ETL engine based on the first set of DSS updates;

automatically determine a second set of DSS updates for a semantic layer of a business intelligence (BI) engine of the DSS based on the change in the set of data sources and based on the first set of DSS updates, wherein the semantic layer comprises a mapping of fields from the data warehouse to reporting fields of the BI engine; and

automatically update the semantic layer of the BI engine based on the second set of DSS updates.