Real time data warehousing

• US 8,452,787 B2
• Filed: 12/27/2002
• Issued: 05/28/2013
• Est. Priority Date: 12/28/2001
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Jonas, Jeffrey James
• Primary Examiner(s): BROWN, SHEREE N
• Application Number:
  US10/331,068
• Publication Number:
  US 20030154194A1
• Time in Patent Office: 3,805 Days
• Field of Search: 707/100, 707/3, 707/10, 707/200, 707/102, 707/758
• US Class Current: 707/758
• CPC Class Codes: G06F 16/20 of structured data, e.g. relational data
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses

Abstract

A method and system for processing data into and in a database and for retrieving the processed data is disclosed. The data comprises identifiers of a plurality of entities. The method and system comprises: (a) processing data into and in a database, (b) enhancing received data prior to storage in a database, (c) determining and matching records based upon relationships between the records in the received data and existing data without any loss of data, (d) enabling alerts based upon user-defined alert rules and relationships, (e) automatically stopping additional matches and separating previously matched records when identifiers used to match records are later determined to be common across entities and not generally distinctive of an entity, (f) receiving data queries for retrieving the processed data stored in the database, (g) utilizing the same algorithm to process the queries and (h) transferring the processed data to another database that uses the same algorithm.

Claims

1. A method for separating previously matched records, the method comprising the steps of:

determining whether a particular identifier in at least one record representing at least one entity is;

(1) common across records representing at least two different entities and (2) no longer generally distinctive of a record representing a single entity;

separating records that were previously matched based on the particular identifier, when the particular identifier is determined to be;

(1) common across records representing at least two different entities and (2) no longer generally distinctive of a record representing a single entity;

storing the separated records;

adding the particular identifier to a list of common identifiers, when the particular identifier is determined to be;

(1) common across records representing at least two different entities and (2) no longer generally distinctive of a record representing a single entity; and

prohibiting any additional matches of records based on the particular identifier when the particular identifier is determined to be (1) common across records representing at least two different entities and (2) no longer generally distinctive of a record representing a single entity.

2. The method of claim 1 further comprising the step of re-submitting the separated records as new received data to be processed.

3. The method of claim 1 wherein the steps of determining whether a particular identifier in at least one record representing at least one entity is (1) common across records representing at least two different entities and (2) no longer generally distinctive of a record representing a single entity and separating records that were previously matched are performed in real-time.

4. The method of claim 1 wherein the steps of determining whether a particular identifier in at least one record representing at least one entity is (1) common across records representing at least two different entities and (2) no longer generally distinctive of a record representing a single entity and separating records that were previously matched are performed in batch.

5. A computer readable medium containing program instructions for execution by a computer for performing a method for separating previously matched records, the method comprising the steps of:

determining whether a particular identifier in at least one record representing at least one entity is;

(1) common across records representing at least two different entities and (2) no longer generally distinctive of a record representing a single entity; and

separating records that were previously matched based on the particular identifier when the particular identifier is determined to be;

(1) common across records representing at least two different entities and (2) no longer generally distinctive of a record representing a single entity;

storing the separated records;

adding the particular identifier to a list of common identifiers, when the particular identifier is determined to be;

(1) common across records representing at least two different entities and (2) no longer generally distinctive of a record representing a single entity; and

prohibiting any additional matches of records based on a particular identifier when the particular identifier is determined to be (1) common across records representing at least two different entities and (2) no longer generally distinctive of a record representing a single entity.

6. The computer readable medium of claim 5 further comprising the step of re-submitting the separated records as new received data to be processed by the system.

7. The computer readable medium of claim 5 wherein the steps of determining whether a particular identifier in at least one record representing at least one entity is (1) common across records representing at least two different entities and (2) no longer generally distinctive of a record representing a single entity and separating records that were previously matched are performed in real-time.

8. The computer readable medium of claim 5 wherein the steps of determining whether a particular identifier in at least one record representing at least one entity is (1) common across records representing at least two different entities and (2) no longer generally distinctive of a record representing a single entity and separating records that were previously matched are performed in batch.

9. An apparatus for separating previously matched records, comprising:

one or more computers; and

one or more processes performed by the one or more computers, the processes configured for;

determining whether a particular identifier in at least one record representing at least one entity is;

(1) common across records representing at least two different entities and (2) no longer generally distinctive of a record representing a single entity;

separating records that were previously matched based on the particular identifier, when the particular identifier is determined to be;

(1) common across records representing at least two different entities and (2) no longer generally distinctive of a record representing a single entity;

storing the separated records;

adding the particular identifier to a list of common identifiers, when the particular identifier is determined to be;

(1) common across records representing at least two different entities and (2) no longer generally distinctive of a record representing a single entity; and

prohibiting any additional matches of records based on the particular identifier when the particular identifier is determined to be (1) common across records representing at least two different entities and (2) no longer generally distinctive of a record representing a single entity.

10. The system of claim 9 wherein the processes are further configured for re-submitting the separated records as new received data to be processed.

11. The system of claim 9 wherein the processes configured for determining whether a particular identifier in at least one record representing at least one entity is (1) common across records representing at least two different entities and (2) no longer generally distinctive of a record representing a single entity and separating records that were previously matched are performed in real-time.

12. The system of claim 9 wherein the processes configured for determining whether a particular identifier in at least one record representing at least one entity is (1) common across records representing at least two different entities and (2) no longer generally distinctive of a record representing a single entity and separating records that were previously matched are performed in batch.