Rapid caching and data delivery system and method

• US 8,032,485 B2
• Filed: 02/01/2008
• Issued: 10/04/2011
• Est. Priority Date: 02/01/2008
• Status: Expired due to Fees

• Current Assignee: Liberty Peak Ventures, LLC (Dominion Harbor Enterprises, LLC)
• Original Assignee: American Express Travel Related Services Company, Inc. (American Express Company)
• Inventors: Thomas, Deep, Khalfin, Marat S, Durvasula, Sastry VSM, Dantu, Stephen K, Alvarez, Eddie J
• Primary Examiner(s): Breene, John E
• Assistant Examiner(s): PHILLIPS, III, ALBERT M
• Application Number:
  US12/024,926
• Publication Number:
  US 20090198892A1
• Time in Patent Office: 1,341 Days
• Field of Search: 707/602, 707/795
• US Class Current: 707/602
• CPC Class Codes: G06F 12/0875 with dedicated cache, e.g. instruction or stack
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses

Abstract

The initial systems analysis of a new data source fully defines each data element and also designs, tests and encodes complete data integration instructions for each data element. A metadata cache stores the data element definition and data element integration instructions. The metadata cache enables a comprehensive view of data elements in an enterprise data architecture. When data is requested that includes data elements defined in a metadata cache, the metadata cache and associated software modules automatically generate database elements to fully integrate the requested data elements into existing databases.

Claims

1. A computer-based method for data integration comprising:

receiving, by a data integration computer, a first request for a first subset comprising a first data element subset and a first raw data subset derived from data, wherein said data includes at least one of raw data and data elements relating to said raw data, and wherein said data includes said first raw data subset and said first data element subset;

receiving, by said computer, said data from a data source;

assessing, by said computer, said data for integration into an enterprise data architecture;

integrating, by said computer, said first subset into said enterprise data architecture based upon said assessing;

creating, by said computer, metadata related to said data and based upon said assessing, wherein said metadata comprises data element definitions associated with said data elements and at least one of data extraction logic and data transformation logic;

caching, by said computer, said data and said metadata to create cached data and cached metadata;

receiving, by said computer, a second request for a second subset comprising a second data element subset and a second raw data subset derived from said data;

analyzing, by said computer and based upon said second data element subset, said enterprise data architecture and said cached metadata to determine a third data element subset comprising data elements present in said cached data but not present in said enterprise data architecture;

integrating, by said computer, said third data element subset into said enterprise data architecture;

analyzing, by said computer and based upon said second raw data, said enterprise data architecture and said cached data to determine third raw data, said third raw data being present in said cached data but not present in said enterprise data architecture;

extracting, by said computer and based upon said data extraction logic, said third raw data; and

integrating, by said computer, said third raw data into said enterprise data architecture.

2. The method of claim 1, wherein said second subset includes a portion of said first subset.

3. The method of claim 1, further comprising transmitting a notification of said step of integrating at least one of said first subset and said second subset.

4. The method of claim 1, wherein creating metadata further comprises creating data transformation scripts and data loading scripts comprising instructions for integrating said data into said enterprise data architecture.

5. The method of claim 1, further comprising packaging said metadata into an envelope.

6. The method of claim 1, further comprising packaging said metadata into an envelope and storing said envelope in a metadata cache.

7. The method of claim 1, further comprising packaging said metadata into an envelope, wherein said metadata comprises at least one of:

said data source, description of a data schema for said data source, data type information, data extraction logic, a data constraint, a validation rule, a default value, a business rule, a data interdependency, data transformation logic, a data load instruction, performance data, statistical data, a business rule, and a physical relationship with said enterprise data schema.

8. The method of claim 1, wherein said second request includes data elements not included in said first request.

9. The method of claim 1, wherein said integrating at least one of:

said first subset and said second subset further comprises creating a database.

10. The method of claim 1, wherein said integrating at least one of:

said first subset and said second subset further comprises creating a database and monitoring said database.

11. The method of claim 1, wherein said integrating at least one of:

said first subset and said second subset further comprises creating a database, monitoring said database to obtain database performance tuning information, and tuning said database based upon said database performance tuning information.

12. The method of claim 1, further comprising transmitting said cached metadata.

13. A non-transitory machine-readable medium having stored thereon a plurality of instructions integrating data, said plurality of instructions when executed by at least one data integration processor, cause said processor to perform operations comprising:

receiving, by said processor, a first request for a first subset comprising a first data element subset and a first raw data subset derived from data, wherein said data includes at least one of raw data and data elements relating to said raw data, and wherein said data includes said first raw data subset and said first data element subset;

receiving, by said processor, said data from a data source;

assessing, by said processor, said data for integration into an enterprise data architecture;

integrating, by said processor, said first subset into said enterprise data architecture based upon said assessing;

creating, by said processor, metadata related to said data and based upon said assessing, wherein said metadata comprises data element definitions associated with said data elements and at least one of data extraction logic and data transformation logic;

caching, by said processor, said data and said metadata to create cached data and cached metadata;

receiving, by said processor, a second request for a second subset comprising a second data element subset and a second raw data subset derived from said data;

analyzing, by said processor and based upon said second data element subset, said enterprise data architecture and said cached metadata to determine a third data element subset comprising data elements present in said cached data but not present in said enterprise data architecture;

integrating, by said processor, said third data element subset into said enterprise data architecture;

analyzing, by said processor and based upon said second raw data, said enterprise data architecture and said cached data to determine third raw data, said third raw data being present in said cached data but not present in said enterprise data architecture;

extracting, by said processor and based upon said data extraction logic, said third raw data; and

,

integrating, by said processor, said third raw data into said enterprise data architecture.

14. A system, comprising:

a network interface communicating with a non-transitory memory;

said memory communicating with a data integration processor; and

said processor, when executing a data integration computer program, performs operations comprising;

receiving, by said processor, a first request for a first subset comprising a first data element subset and a first raw data subset derived from data, wherein said data includes at least one of raw data and data elements relating to said raw data, and wherein said data includes said first raw data subset and said first data element subset;

receiving, by said processor, said data from a data source;

assessing, by said processor, said data for integration into an enterprise data architecture;

integrating, by said processor, said first subset into said enterprise data architecture based upon said assessing;

creating, by said processor, metadata related to said data and based upon said assessing, wherein said metadata comprises data element definitions associated with said data elements and at least one of data extraction logic and data transformation logic;

caching, by said processor, said data and said metadata to create cached data and cached metadata;

receiving, by said processor, a second request for a second subset comprising a second data element subset and a second raw data subset derived from said data;

analyzing, by said processor and based upon said second data element subset, said enterprise data architecture and said cached metadata to determine a third data element subset comprising data elements present in said cached data but not present in said enterprise data architecture;

integrating, by said processor, said third data element subset into said enterprise data architecture;

analyzing, by said processor and based upon said second raw data, said enterprise data architecture and said cached data to determine third raw data, said third raw data being present in said cached data but not present in said enterprise data architecture;

extracting, by said processor and based upon said data extraction logic, said third raw data; and

,

integrating, by said processor, said third raw data into said enterprise data architecture.