DATA WAREHOUSE COMPATIBILITY

• US 20140358845A1
• Filed: 06/03/2013
• Published: 12/04/2014
• Est. Priority Date: 06/03/2013
• Status: Active Grant

• Current Assignee: Bank of America Corp.
• Original Assignee: Bank of America Corp.
• Inventors: Mundlapudi, Bharath, Banala, Karthik, Koneru, Rajesh
• Granted Patent: US 9,460,188 B2
• Time in Patent Office: Days
• US Class Current: 707/602
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
• CPC Class Codes: G06F 16/283 Multi-dimensional databases or data warehouses, e.g. MOLAP or ROLAP

Abstract

A compatibility processing module, for executing one or more processes to format and manipulate data, such that communication between previously-incompatible data warehouses is facilitated. In particular, a first warehouse is disclosed, wherein the first data warehouse is configured with a compatibility processing module, for receiving a large number of data points, and for executing one or more processes on a stored portion of the received data points such that the resulting processed data points are compatible with formatting conventions of a second data warehouse.

Claims

1. A system for establishing compatibility between an open-source data warehouse and a proprietary data warehouse, the system comprising:

a distribution processing module configured to receive a data stream;

a distributed file system configured to store at least a portion of the data stream according to a first data format; and

a first data warehouse configured to an execute extract, transform, and load (ETL) operation on the stored portion of the data stream;

wherein the first data warehouse includes a compatibility processing module configured to execute one or more transformation processes on the stored portion of the data stream; and

wherein the compatibility processing module formats the stored portion of the data stream according to a second data format such that the stored portion of the data stream is compatible with a second data warehouse that uses the second data format.

2. The system according to claim 1, wherein the data stream comprises raw data.

3. The system according to claim 1, wherein the compatibility processing module executes a precision-adjustment process that rounds numerical values according to one or more standards of numerical precision associated with the second data format.

4. The system according to claim 1, wherein the compatibility processing module executes a data type conversion process that converts first data having a first data type stored in the first data warehouse into second data having a second data type recognized by the second data warehouse.

5. The system according to claim 1, wherein the compatibility processing module executes a single rank process to rank a data set comprising data from the data stream, wherein the single rank process ranks the data set based on a single metric.

6. The system according to claim 1, wherein the compatibility processing module executes a group rank process to rank a data set comprising data from the data stream, wherein the group rank process ranks the data set based two or more metrics.

7. The system according to claim 1, wherein the compatibility processing module executes a date format process that converts first data having a first date format into second data having a second date format recognized by the data warehouse.

8. The system according to claim 1, further comprising a distributed non-relational database that stores the stored portion of the data stream.

9. The system according to claim 1, further comprising a parallel processing module, for dividing processes to be executed on the data stream among a plurality of distributed computers.

10. A non-transitory computer-readable storage medium having computer-executable program instructions stored thereon that when executed by a processor cause the processor to perform steps establishing compatibility between a first data warehouse and a second data warehouse, the steps comprising:

receiving, from a distribution processing module, a data stream;

storing, in a distributed file system, at least a portion of the data stream according to a first data format;

executing, using the first data warehouse, an extract, transform, and load (ETL) operation on the stored portion of the data stream; and

executing, using a compatibility processing module, one or more transformation processes on the stored portion of the data stream, according to a second data format such that the stored portion of the data stream is compatible with a second data warehouse that uses the second data format.

11. The computer-readable storage medium of claim 10, wherein the data stream comprises raw data.

12. The computer-readable storage medium of claim 10, wherein the computer-executable instructions, when executed by the processor, cause the processor to perform steps further comprising executing, by the compatibility processing module, a precision-adjustment process to round off numerical values according to one or more standards of numerical precision associated with the proprietary data warehouse.

13. The computer-readable storage medium of claim 10, wherein the computer-executable instructions, when executed by the processor, cause the processor to perform steps further comprising executing, by the compatibility processing module, a data type conversion process that converts first data having a first data type stored in the data warehouse into second data having a second data type recognized by the second data warehouse.

14. The computer-readable storage medium of claim 10, wherein the computer-executable instructions, when executed by the processor, cause the processor to perform steps further comprising executing, by the compatibility processing module, a single rank process to rank a data set comprising data from the data stream, wherein the single rank process ranks the data set based on a single metric.

15. The computer-readable storage medium of claim 10, wherein the computer-executable instructions, when executed by the processor, cause the processor to perform steps further comprising executing, by the compatibility processing module, a group rank process to rank a data set comprising data from the data stream, wherein the group rank process ranks the data set based two or more metrics.

16. The computer-readable storage medium of claim 10, wherein the computer-executable instructions, when executed by the processor, cause the processor to perform steps further comprising executing, by the compatibility processing module, a date format process that converts first data having a first date format into second data having a second date format recognized by the data warehouse.

17. The computer-readable storage medium of claim 10, wherein the computer-executable instructions, when executed by the processor, cause the processor to perform steps further comprising storing, in a distributed non-relational database, the stored portion of the data stream.

18. The computer-readable storage medium of claim 10, wherein the computer-executable instructions, when executed by the processor, cause the processor to perform steps further comprising dividing, using a parallel processing module, processes to be executed on the data stream among a plurality of distributed computers.

19. A method of establishing compatibility between a first data warehouse and a second data warehouse comprising:

receiving a data stream, by a distribution processing module, wherein the data stream comprises raw data;

storing at least a portion of the data stream in a distributed file system according to a first data format;

executing an extract, transform, and load (ETL) operation on the stored portion of the data stream;

executing one or more transformation processes on the stored portion of the data stream; and

formatting the stored portion of the data stream according to a second data format such that the stored portion of the data stream is compatible with a second data warehouse that uses the second data format.

20. The method of claim 19 wherein:

the first data warehouse is an open-source data warehouse;

the second data warehouse is a commercially available data warehouse; and

the distributed file system is an open-source distributed file system;