Collaborative data intelligence between data warehouse models and big data stores

• US 10,127,293 B2
• Filed: 03/30/2015
• Issued: 11/13/2018
• Est. Priority Date: 03/30/2015
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Chode, Bibekananda C., Kallur, Chandra S. R., Kanamatareddy, Ravi Kumar Reddy, Vijayan, Vinoth
• Primary Examiner(s): Allen, Brittany N
• Assistant Examiner(s): Bartlett, William P
• Application Number:
  US14/672,304
• Publication Number:
  US 20160292256A1
• Time in Patent Office: 1,324 Days
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor

Abstract

Embodiments of the present invention provide a system and method for collaborative data intelligence. Embodiments of the present invention can include one or more data stores and one or more data warehouses (DWHs), each individually configured to communicate with a physical storage management system. Embodiments of the present invention can be used to manage the chaotic disorder of social data in real time, using a collaborative DWH model and big data store, in order to assess the potential of data integration to reporting and decision management.

Claims

1. A collaborative data intelligence system, the system comprising:

a data store and a data warehouse (DWH), wherein the data store and the DWH are configured to communicate with a storage management system;

an integration bus, wherein the integration bus comprises at least one data analysis module;

a decision process application, wherein the decision process application comprises a result after processing data associated with the data store and the DWH;

one or more computer processors;

one or more computer-readable storage media;

program instructions stored on the computer-readable storage media for execution by at least one of the one or more computer processors, the program instructions comprising;

program instructions to receive a plurality of data that can either be structured or unstructured from a data store storing web content;

program instructions to receive, from a data warehouse (DWH), a plurality of structured data;

program instructions to facilitate communications of the plurality of structured data from the DWH and the plurality of data from the data store in a bi-directional manner using proxy caching for data delivery, wherein the program instructions to facilitate communications comprises;

program instructions to standardize the plurality of data and the plurality of structured data with respect to either the plurality of data from the data store or the plurality of structured data received from the DWH using a set of integration calculations associated with the data store and the DWH, wherein the standardized plurality of data and plurality of structured data perform a hybrid management of workloads of the DWH and the data store, wherein the set of integration calculations comprise;

program instructions to schedule workloads of the DWH and the data store based on task composition with respect to either the DWH or the data store in the bi-directional manner, and

program instructions to maintain integrity of synchronization of data between the DWH and the data store;

program instructions to correct the communications between the DWH and the data store;

program instructions to update in real time the standardized plurality of data and plurality of structured data; and

program instructions to generate a decision report for the standardized plurality of data and plurality of structured data.

2. The system of claim 1, further comprising:

at least one application, wherein the at least one application is configured to communicate with the integration bus.

3. The system of claim 1, wherein the at least one data analysis module comprises:

an integration module, a data acceptability module, a modeling module, a corrective module, and a directional module.

4. The system of claim 1, wherein the at least one data analysis module is configured to include at least one aspect of integration between the DWH and the data store, and wherein the at least one aspect of integration comprises:

master data, pre-aggregated data, post-aggregated data, business reference details, and interpreted models.

5. The system of claim 1, wherein the at least one data analysis module is configured with predetermined integration options.

6. The system of claim 1, wherein the at least one data analysis module is configured to determine a level of acceptability of data associated with the data store.

7. The system of claim 1, wherein the at least one data analysis module is configured to identify a set of ordinals from a set of cardinals included in a plurality of source data.

8. The system of claim 1, wherein the at least one data analysis module is configured to identify corrections to processed data associated with the DWH and the data store.

9. The system of claim 1, wherein the data store is configured to receive unstructured data, and wherein the unstructured data comprises:

web content, e-mails, and social media data.

10. The system of claim 1, wherein the result after processing the data associated with the data store and the DWH comprises a combined report.

11. The system of claim 1, wherein the storage management system is configured to maintain a physical store of the data associated with the data store and the DWH.

12. A method for analyzing data, the method comprising:

receiving, by an integration bus, a plurality of data that can either be structured or unstructured from a data store storing web content, wherein the integration bus comprises at least one data analysis module;

receiving, by the integration bus, from a data warehouse (DWH), a plurality of structured data;

facilitating communications of the plurality of structured data from the DWH and the plurality of data from the data store in a bi-directional manner using proxy caching for data delivery, wherein the program instructions to facilitate communications comprises;

standardizing the plurality of data and the plurality of structured data with respect to either the plurality of data from the data store or the plurality of structured data received from the DWH using a set of integration calculations associated with the data store and the DWH, wherein the standardized plurality of data and plurality of structured data perform a hybrid management of workloads of the DWH and the data store, wherein the set of integration calculations comprises;

scheduling workloads of the DWH and the data store based on task composition with respect to either the DWH or the data store in the bi-directional manner, and

maintaining integrity of synchronization of data between the DWH and the data store;

correcting the communications between the DWH and the data store;

updating in real time, by the at least one data analysis module, the standardized plurality of data and plurality of structured data; and

generating, by the at least one data analysis module, a decision report for the standardized plurality of data and plurality of structured data.

13. The method of claim 12, wherein the decision report comprises information associated with the set of integration calculations made by the at least one data analysis module.

14. The method of claim 13, wherein the decision report comprises a combined report from the data store and the DWH.

15. The method of claim 12, wherein the at least one data analysis module comprises:

an integration module, a data acceptability module, a modeling module, a corrective module, and a directional module.

16. The method of claim 12, further comprising:

receiving, by the data store, unstructured data, wherein the unstructured data comprises web content, e-mails, and social media data.

17. The method of claim 12, wherein performing, by the at least one data analysis module of the integration bus, a set of integration calculations associated with the data store and the DWH comprises:

determining, by the at least one data analysis module, a level of acceptability of data associated with the data store;

identifying, by the at least one data analysis module, a set of ordinals from a set of cardinals included in a plurality of source data; and

identifying, by the at least one data analysis module, corrections to processed data associated with the DWH and the data store.

18. The method of claim 12, wherein the set of integration calculations are performed simultaneously.

19. A computer program product comprising:

a computer readable storage medium and program instructions stored on the computer readable storage medium for execution by at least one or more processors, the program instructions comprising;

program instructions to receive a plurality of data that can either be structured or unstructured from a data store storing web content

program instructions to receive, from a data warehouse (DWH), a plurality of structured data;

program instructions to facilitate communications of the plurality of structured data from the DWH and the plurality of data from the data store in a bi-directional manner using proxy caching for data delivery, wherein the program instructions to facilitate communications comprises;

program instructions to standardize the plurality of data and the plurality of structured data with respect to either the plurality of data from the data store or the plurality of structured data received from the DWH using a set of integration calculations associated with the data store and the DWH, wherein the standardized plurality of data and plurality of structured data perform a hybrid management of workloads of the DWH and the data store, wherein the set of integration calculations comprise;

program instructions to schedule workloads of the DWH and the data store based on task composition with respect to either the DWH or the data store in the bi-directional manner, and

program instructions to maintain integrity of synchronization of data between the DWH and the data store;

program instructions to correct the communications between the DWH and the data store;

program instructions to update in real time the standardized plurality of data and plurality of structured data; and

program instructions to generate a decision report for the standardized plurality of data and plurality of structured data.

20. The computer program product of claim 19, wherein the at least one data analysis module comprises:

an integration module, a data acceptability module, a modeling module, a corrective module, and a directional module.