Method and apparatus for data integration and management

• US 8,166,048 B2
• Filed: 01/13/2009
• Issued: 04/24/2012
• Est. Priority Date: 06/27/2005
• Status: Active Grant

• Current Assignee: Informatica Corporation
• Original Assignee: Informatica Corporation
• Inventors: Wong, Stanley, Hoskin, Steve, Oian, Xiao Feng
• Primary Examiner(s): PARDO, THUY N
• Application Number:
  US12/353,062
• Publication Number:
  US 20090182780A1
• Time in Patent Office: 1,197 Days
• Field of Search: 707/638, 707/652, 707/735, 707/748, 707/758, 707/769, 705/1, 705/36
• US Class Current: 707/748
• CPC Class Codes: G06F 16/25 Integrating or interfacing systems involving database management systems
• CPC Class Codes: G06Q 30/02 Marketing; Price estimation or determination; Fundraising
• CPC Class Codes: Y10S 707/99939 Privileged access
• CPC Class Codes: Y10S 707/99942 Manipulating data structure, e.g. compression, compaction, compilation
• CPC Class Codes: Y10S 707/99943 Generating database or data structure, e.g. via user interface
• CPC Class Codes: Y10S 707/99944 Object-oriented database structure
• CPC Class Codes: Y10S 707/99945 Object-oriented database structure processing

Abstract

A system for maintaining master reference data for entities is disclosed. The system includes multiple reference data sets at multiple different data storages. A reference data set for a particular entity uniquely identifies the particular entity. The system also includes a first master reference data set at a first data storage that is at least as reliable as a second reference data set at a second data storage. In some embodiments, the first data storage can be updated through real-time process or an offline process (e.g., a batch process). The first master reference data set includes at least one data record, and content metadata regarding the data record. In some embodiments, content metadata comprises lineage data that includes each preceding value that was contained in the data record. In some of these embodiments, lineage includes other factors that affected the present and previous values contained in the data record.

Claims

1. A system for maintaining master reference data relating to entities, said system comprising:

a master reference data storage for storing a plurality of master reference data sets based on reference data sets from different data storages, wherein each reference data set identifies a particular entity, wherein each of a plurality of the entities is identified by a plurality of reference data sets stored in a plurality of the data storages, wherein a master reference data set for a particular entity represents a most reliable reference data set stored for the particular entity in any one of the different data storages; and

a master reference manager for receiving a reference data set in real time, updating the master reference data set in real time based on the received reference data set, and maintaining content metadata for the master reference data set, wherein the master reference manager includes a trust framework for updating the master reference data set in real time, wherein the master reference data set includes a set of data fields, wherein updating the master reference data set comprises changing a current value of a data field of the master reference data set when the reference data set includes another value for the data field that is more reliable than the current value of the data field.

2. The system of claim 1, wherein the content metadata includes lineage data or historical data for maintaining the reliability of the master reference data set, said lineage data or historical data comprising a previous value for the master reference data set.

3. The system of claim 1, wherein the trust framework includes a trust score calculator that calculates at least one trust score for the reference data set, wherein the master reference manager determines that the reference data set'"'"'s other value for the data field is more reliable than the current value by comparing the trust score with at least one other trust score calculated for the master reference data set.

4. The system of claim 1, wherein the master reference manager maintains the content metadata by modifying the content metadata of the master reference data set to include a source of the reference data set when updating the master reference data set.

5. The system of claim 1, wherein the trust framework is further for merging, periodically or in real time, two master reference data sets that relate to a same entity.

6. The system of claim 1, wherein the trust framework includes:

a trust score calculator for calculating at least one trust score for data values of the master reference data set; and

a set of configurable rules by which the trust score calculator calculates the at least one trust score for the data values of the master reference data set.

7. The system of claim 6, wherein the set of configurable rules include at least one of (i) a first set of rules specifying a reliability of a particular value in a particular data set, (ii) a second set of rules specifying how the reliability of the particular value changes over time based on a date that the particular value was updated, and (iii) a third set of rules that specify how the reliability of the particular value is modified based on one or more formulas used to validate a particular type of data.

8. A system for maintaining master reference data relating to entities, said system comprising:

a master reference data storage for storing a plurality of master reference data sets based on reference data sets from different data storages, wherein each reference data set identifies a particular entity, wherein each of a plurality of the entities is identified by a plurality of reference data sets stored in a plurality of the data storages, wherein a master reference data set for a particular entity represents a most reliable reference data set stored for the particular entity in any one of the different data storages; and

a master reference manager for receiving a reference data set in real time, merging the master reference data set in real time based on the received reference data set, and maintaining content metadata for the master reference data set, wherein the master reference manager includes a trust framework for merging the master reference data set in real time, wherein the master reference data set includes a set of data fields, wherein merging the master reference data set comprises changing a current value of a data field of the master reference data set when the master reference manager identifies that the reference data set includes another value for the data field that is more reliable than the current value of the data field in the master reference data set.

9. The system of claim 8, wherein the trust framework is further for updating the master reference data set in real time.

10. The system of claim 8, wherein the trust framework includes a trust score calculator that calculates at least one trust score for the reference data set, wherein the master reference manager determines that the reference data set'"'"'s other value for the data field is more reliable than the current value by comparing the trust score with at least one other trust score calculated for the master reference data set.

11. The system of claim 8, wherein the master reference manager modifies the content metadata of the master reference data set when merging the master reference data set with the reference data set.

12. The system of claim 8, wherein the master reference manager merges two master reference data sets upon determining that the two master reference data sets includes a set of data values that match.

13. A system for maintaining master reference data relating to entities, said system comprising:

a master reference data storage for storing a plurality of master reference data sets based on reference data sets from different data storages, wherein each reference data set identifies a particular entity, wherein each of a plurality of the entities is identified by a plurality of reference data sets stored in a plurality of the data storages, wherein a master reference data set for a particular entity represents a most reliable reference data set stored for the particular entity in any one of the different data storages; and

a master reference manager for receiving a reference data set in real time, modifying the master reference data set in real time based on the received reference data set, and maintaining content metadata for the master reference data set, wherein the master reference manager includes a trust framework for merging two master reference data sets in real time, wherein the master reference manager merges the two master reference data sets by determining whether the two master reference data sets include a first set of data values that match, wherein when the two master reference data sets do not include the first set of data values that match but include a second set of data values that match, the master reference manager identifies the two master reference data sets as potentially matching master reference data sets that should be examined by a system operator to determine whether the two master reference data sets are matching and need to be merged.

14. The system of claim 13, wherein the master reference manager maintains the content metadata by modifying the content metadata of the master reference data set to include at least one of a previous value of the master reference data set and a source of the reference data set.

15. A system for maintaining master reference data relating to entities, said system comprising:

a master reference data storage for storing a plurality of master reference data sets based on reference data sets from different data storages, wherein each reference data set identifies a particular entity, wherein each of a plurality of the entities is identified by a plurality of reference data sets stored in a plurality of the data storages, wherein a master reference data set for a particular entity represents a most reliable reference data set stored for the particular entity in any one of the different data storages; and

a master reference manager for receiving a reference data set in real time, modifying the master reference data set in real time based on the received reference data set, and maintaining content metadata for the master reference data set, wherein the master reference manager includes a trust framework for merging first and second master reference data sets into a third master reference data set, and performing an unmerge operation that divides the third master reference data set into the first and second master reference data sets that at one time were merged to produce the third master reference data set.

16. The system of claim 15, wherein performing the unmerge operation comprises using content metadata of the third master reference data set to to divide the third master reference data set.

17. The system of claim 15, wherein the master reference manager performs the unmerge operation after modification to a set of matching criteria that is used to merge two or more master reference data sets.

18. A method of maintaining master reference data relating to entities, the method comprising:

storing a plurality of master reference data sets based on reference data sets from different data storages, wherein each of a plurality of the entities is identified by a plurality of reference data sets stored in a plurality of the data storages, wherein a master reference data set for a particular entity represents a most reliable reference data set stored for the particular entity in any one of the different data storages;

receiving a reference data set in real time

updating the master reference data set in real time based on the received reference data set, wherein the master reference data set includes a set of data fields, wherein updating the master reference data set comprises changing a current value of a data field of the master reference data set when the reference data set includes another value for the data field that is more reliable than the current value of the data field; and

maintaining content metadata for the master reference data set.

19. The method of claim 18, wherein updating the master reference data set comprises calculating at least one trust score for the reference data set and determining whether the reference data set'"'"'s other value for the data field is more reliable than the current value by comparing the trust score with at least one other trust score calculated for the master reference data set.

20. The method of claim 18 further comprising performing a match and merge operation that merges, in real-time or periodically, first and second master reference data sets into one merged master reference data set upon determining that the first and second master reference data sets relate to a same entity.

21. The method of claim 20 further comprising:

maintaining content metadata for the merged master reference data set, said content metadata comprising a previous value of the merged master reference data set; and

performing an unmerge operation that restores the first and second master reference data sets based on the previous value included in the content metadata.

22. The method of claim 18, wherein maintaining the content metadata comprises updating the content metadata to include at least one of a previous value of the master reference data set and a source of the reference data set.