Rapid caching and data delivery system and method

• US 8,195,603 B2
• Filed: 09/01/2011
• Issued: 06/05/2012
• Est. Priority Date: 02/01/2008
• Status: Expired due to Fees

• Current Assignee: Liberty Peak Ventures, LLC (Dominion Harbor Enterprises, LLC)
• Original Assignee: American Express Travel Related Services Company, Inc. (American Express Company)
• Inventors: Thomas, Deep, Khalfin, Marat S., Durvasula, Sastry VSM, Dantu, Stephen K., Alvarez, Eddie J.
• Primary Examiner(s): Breene, John E
• Assistant Examiner(s): PHILLIPS, III, ALBERT M
• Application Number:
  US13/224,184
• Publication Number:
  US 20110320506A1
• Time in Patent Office: 278 Days
• Field of Search: 707/602, 707/795
• US Class Current: 707/602
• CPC Class Codes: G06F 12/0875 with dedicated cache, e.g. instruction or stack
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses

Abstract

The initial systems analysis of a new data source fully defines each data element and also designs, tests and encodes complete data integration instructions for each data element. A metadata cache stores the data element definition and data element integration instructions. The metadata cache enables a comprehensive view of data elements in an enterprise data architecture. When data is requested that includes data elements defined in a metadata cache, the metadata cache and associated software modules automatically generate database elements to fully integrate the requested data elements into existing databases.

Claims

1. A computer-based method comprising:

analyzing, by a data integration computer and based upon a first data element subset, enterprise data architecture and cached metadata to determine a second data element subset comprising data elements present in said cached data but not present in said enterprise data architecture, wherein said cached metadata is derived from caching metadata;

integrating, by said computer, a said second data element subset into said enterprise data architecture;

analyzing, by said computer and based upon first raw data, said enterprise data architecture and said cached data to determine second raw data, said second raw data being present in said cached data but not present in said enterprise data architecture;

extracting, by said computer and based upon data extraction logic, said second raw data; and

integrating, by said computer, said second raw data into said enterprise data architecture.

2. The method of claim 1, further comprising caching, by said computer, said enterprise data architecture and said metadata to create said cached enterprise data architecture and said cached metadata.

3. The method of claim 1, wherein said metadata comprises data element, definitions associated with data elements and at least one of data extraction logic and data transformation logic.

4. The method of claim 1, wherein a first subset is integrated into said enterprise data architecture.

5. The method of claim 4, wherein a second subset includes a portion of said first subset.

6. The method of claim 5, wherein said first subset comprises a first data element subset and a first raw data subset derived from first data.

7. The method of claim 6, wherein said first data includes at least one of raw data and data elements relating to said raw data.

8. The method of claim 6, wherein said first data includes a first raw data subset and a first data element subset.

9. The method of claim 5, wherein said second subset comprises a second data element subset and a second raw data subset derived from said first data.

10. The method of claim 5, further comprising transmitting a notification of said integrating at least one of said first subset and said second subset.

11. The method of claim 5, wherein said integrating at least one of:

said first subset and said second subset further comprises creating a database.

12. The method of claim 5, wherein said integrating at least one of:

said first subset and said second subset further comprises creating a database and monitoring said database.

13. The method of claim 5, wherein said integrating at least one of:

said first subset and said second subset further comprises creating a database, monitoring said database to obtain database performance tuning information, and tuning said database based upon said database performance tuning information.

14. The method of claim 1, further comprising creating metadata by creating data transformation scripts and data loading, scripts comprising instructions for integrating first data into said enterprise data architecture.

15. The method of claim 1, further comprising packaging said metadata into an envelope.

16. The method of claim 1, further comprising packaging said metadata into an envelope and storing said envelope in a metadata cache.

17. The method of claim 1, further comprising packaging said metadata into an envelope, wherein said metadata comprises at least one of:

a data source, description of a data schema for said data source, data type information, data extraction logic, a data constraint, a validation rule, a default value, a business rule, a data interdependency, data transformation logic, a data load instruction, performance data, statistical data, a business rule, and a physical relationship with said enterprise data schema.

18. The method of claim 1, wherein a second request includes data elements not included in a first request.

19. An article of manufacture including a non-transitory, tangible computer readable storage medium having instructions stored thereon that, in response to execution by a computer-based data integration system, cause said computer-based system to perform operations comprising:

analyzing, by said computer-based system and based upon a first data element subset, enterprise data architecture and cached metadata to determine a second data element subset comprising data elements present in said cached data but not present in said enterprise data architecture, wherein said cached metadata is derived from caching metadata;

integrating, by said computer-based system, a said second data element subset into said enterprise data architecture;

analyzing, by said computer-based system and based upon first raw data, said enterprise data architecture and said cached data to determine second raw data, said second raw data being present in said cached data but not present in said enterprise data architecture;

extracting, by said computer-based system and based upon data extraction logic, said second raw data; and

integrating, by said computer-based system, said second raw data into said enterprise data architecture.

20. A system comprising:

a processor for integrating data,

a tangible, non-transitory memory configured to communicate with said processor,

said tangible, non-transitory memory having instructions stored thereon that, in response to execution by said processor, cause said processor to perform operations comprising;

analyzing, by said processor and based upon a first data element subset, enterprise data architecture and cached metadata to determine a second data element subset comprising data elements present in said cached data but not present in said enterprise data architecture, wherein said cached metadata is derived from caching metadata;

integrating, by said processor, a said second data element subset into said enterprise data architecture;

analyzing, by said processor and based upon first raw data, said enterprise data architecture and said cached data to determine second raw data, said second raw data being present in said cached data but not present in said enterprise data architecture;

extracting, by said processor and based upon data extraction logic, said second raw data; and

integrating, by said processor, said second raw data into said enterprise data architecture.