Real time data warehousing

• US 20030154194A1
• Filed: 12/27/2002
• Published: 08/14/2003
• Est. Priority Date: 12/28/2001
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Jonas, Jeffrey James
• Granted Patent: US 8,452,787 B2
• Time in Patent Office: Days
• US Class Current: 707/3
• CPC Class Codes: G06F 16/20 of structured data, e.g. relational data
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses

Abstract

A method and system for processing data into and in a database and for retrieving the processed data is disclosed. The data comprises identifiers of a plurality of entities. The method and system comprises: (a) processing data into and in a database, (b) enhancing received data prior to storage in a database, (c) determining and matching records based upon relationships between the records in the received data and existing data without any loss of data, (d) enabling alerts based upon user-defined alert rules and relationships, (e) automatically stopping additional matches and separating previously matched records when identifiers used to match records are later determined to be common across entities and not generally distinctive of an entity, (f) receiving data queries for retrieving the processed data stored in the database, (g) utilizing the same algorithm to process the queries and (h) transferring the processed data to another database that uses the same algorithm.

Claims

1. A method for processing data comprising the steps of:

receiving data comprising at least one record having at least one identifier, each record representing at least one of a plurality of entities;

utilizing an algorithm to process the received data;

storing the processed data in a database;

receiving data queries for retrieving at least a portion of the data stored in the database; and

utilizing the algorithm to process the queries.

2. The method of claim 1 wherein the entities are people.

3. The method of claim 1 wherein the entities are personal property.

4. The method of claim 3 wherein the personal property are vehicles.

5. The method of claim 1 wherein the entities are real property.

6. The method of claim 1 wherein the entities are organizations.

7. The method of claim 1 wherein the entities are chemical compounds.

8. The method of claim 1 wherein the entities are organic compounds.

9. The method of claim 1 wherein the entities are proteins.

10. The method of claim 1 wherein the entities are biological structures.

11. The method of claim 1 wherein the entities are biometric values.

12. The method of claim 1 wherein the entities are atomic structures.

13. The method of claim 1 further comprising the step of converting the received data into a standardized message format prior to utilizing an algorithm to process the received data.

14. The method of claim 1 wherein the step of utilizing an algorithm to process the received data includes retaining an attribution of each record.

15. The method of claim 14 wherein the step of retaining an attribution of each record includes retaining an identity of:

a source system providing each record and

a unique identifier representing each record in the source system.

16. The method of claim 14 wherein the step of retaining an attribution of each record includes retaining an identity of a query system and a particular user.

17. The method of claim 1 wherein the step of utilizing an algorithm to process the received data includes analyzing the received data prior to one of storage in the database and query in the database.

18. The method of claim 17 wherein the step of analyzing the received data prior to one of storage in the database and query in the database includes comparing at least one of the identifiers against one of:

a user-defined criterion and

at least one data set in one of a secondary database and a list.

19. The method of claim 18 wherein the compared identifier is a name of at least one of the plurality of entities and the data set is in a names root list.

20. The method of claim 18 wherein the compared identifier is an address of at least one of the plurality of entities and the data set is in an address list.

21. The method of claim 18 wherein the step of comparing at least one of the identifiers against a user-defined criterion includes formatting at least one identifier in accordance with the user-defined standard.

22. The method of claim 18 wherein the step of analyzing the received data prior to one of storage in the database or query in the database includes enhancing the received data.

23. The method of claim 22 wherein the step of enhancing the received data includes:

querying at least one data set in one of the secondary database and the list for additional identifiers for the received data, and

supplementing the received data with the additional identifiers.

24. The method of claim 23 wherein the step of querying at least one data set includes:

at least one data set being in the secondary database utilizing the algorithm to query additional databases to locate additional identifiers relating to at least one of the received identifiers; and

supplementing the received data with the additional identifiers located in the secondary database.

25. The method of claim 17 wherein the step of analyzing the received data prior to one of storage in the database and query in the database includes creating hash keys of the identifiers.

26. The method of claim 1 wherein the step of utilizing an algorithm to process received data includes storing in the database processed queries based upon a user-defined criterion.

27. The method of claim 26 wherein the user-defined criterion includes an expiration date.

28. The method of claim 1 wherein the steps of receiving data comprising at least one record having at least one identifier, each record representing at least one of a plurality of entities, utilizing an algorithm to process the received data, and storing the processed data in a database are performed in real-time.

29. The method of claim 1 wherein the steps of receiving data comprising at least one record having at least one identifier, each record representing at least one of a plurality of entities, utilizing an algorithm to process the received data, and storing the processed data in a database are performed in batch.

30. The method of claims 1 or 17, wherein the step of utilizing an algorithm to process the received data includes:

retrieving from the database a group of additional records having identifiers similar to the identifiers in the received data;

analyzing each identifier of the retrieved group of records for a match to at least a portion of the received data;

matching at least a portion of the received data with at least one analyzed record of the retrieved group of records that is determined to reflect a record having identifiers representing an identical one of the plurality of entities;

analyzing whether at least one identifier is included in the at least a portion of the received data that was not previously stored in the at least one analyzed record of the retrieved group of records that is determined to reflect a record having identifiers representing an identical one of the plurality of entities; and

re-analyzing each identifier of the retrieved group of records for a match to;

at least a portion of the received data and

the analyzed record of the retrieved group of records that is determined to reflect a record having identifiers representing an identical one of the plurality of entities; and

storing the matched records in the database.

31. The method of claim 30 wherein matching at least a portion of the received data with at least one analyzed record includes assigning a persistent key.

32. The method of claim 30 wherein the step of utilizing an algorithm to process the received data further comprises retrieving from the database an additional group of records having identifiers similar to the identifiers in:

at least a portion of the received data and

the analyzed record of the retrieved group of records that is determined to reflect a record having identifiers representing an identical one of the plurality of entities prior to re-analyzing each identifier of the retrieved group of records for a match.

33. The method of claim 32 wherein utilizing an algorithm to process the received data includes repeating:

retrieving from the database a group of records;

analyzing each identifier of the retrieved group of records;

matching at least a portion of the received data;

analyzing whether at least one identifier is included in the at least a portion of the received data that was not previously stored;

retrieving from the database an additional group of records; and

re-analyzing each identifier of the retrieved group of records for a match until no additional matches are determined.

34. The method of claim 30 wherein the step of utilizing an algorithm to process the received data includes:

determining whether a particular identifier is one of;

common across records representing at least two different entities and generally distinctive of a record representing a particular entity; and

separating records that were previously matched based on a particular identifier if the particular identifier is determined to be common across records representing at least two different entities and not generally distinctive of a record representing a particular entity.

35. The method of claim 34 wherein the step of utilizing an algorithm to process the received data includes prohibiting any additional matches of records based on a particular identifier if the particular identifier is determined to be common across records representing at least two different entities and not generally distinctive of a record representing a particular entity.

36. The method of claim 34 wherein the step of utilizing an algorithm to process the received data includes re-processing the separated records as received data.

37. The method of claim 34 wherein the steps of determining whether a particular identifier is one of common across records representing at least two different entities and generally distinctive of a record representing a particular entity and separating records that were previously matched are performed in real-time.

38. The method of claim 34 wherein the steps of determining whether a particular identifier is one of common across records representing at least two entities and generally distinctive of a record representing a particular entity and separating records that were previously matched are performed in batch.

39. The method of claim 30 wherein the step of utilizing an algorithm to process the received data includes:

comparing the received data with at least one stored record to determine the existence of a relationship; and

creating a relationship record for each stored record determined to reflect a relationship with at least a portion of the received data.

40. The method of claim 39 wherein the step of utilizing an algorithm to process the received data includes creating at least one confidence indicator for each relationship record.

41. The method of claim 40 wherein the steps of comparing the received data, creating a relationship record, and creating at least one confidence indicator are performed in real-time.

42. The method of claim 40 wherein the steps of comparing the received data, creating a relationship record, and creating at least one confidence indicator are performed in batch.

43. The method of claim 40 wherein at least one of the confidence indicators indicates the likelihood of a relationship between:

an entity represented by the particular record having a relationship with the portion of the received data, and

an entity represented by the portion of the received data.

44. The method of claim 40 wherein at least one of the confidence indicators indicates the likelihood that:

an entity represented by the particular record having a relationship with the portion of the received data, and

an entity represented by the portion of the received data are the same.

45. The method of claim 40 wherein the step of utilizing an algorithm to process received data includes analyzing the relationship records to determine whether the relationship records reflect at least one relationship not previously determined.

46. The method of claim 45 wherein the step of analyzing the relationship records includes analyzing relationship records reflecting at least one level of degrees of separation.

47. The method of claim 46 wherein the step of analyzing relationship records reflecting at least one level of degrees of separation includes analyzing relationship records meeting at least one user-defined criterion.

48. The method of claim 47 wherein the step of analyzing relationship records meeting at least one user-defined criterion includes limiting the relationship records analyzed to a maximum level of degrees of separation.

49. The method of claim 47 wherein the step of analyzing relationship records meeting at least one user-defined criterion includes limiting the relationship records analyzed to relationship records that include confidence indicators greater than a minimum amount.

50. The method of claim 45 wherein the step of utilizing an algorithm to process received data further comprises issuing an alert based upon at least one user-defined alert rule.

51. The method of claim 50 wherein the step of issuing an alert based upon at least one user-defined alert rule includes having the alert communicated via electronic communications means.

52. The method of claim 51 wherein the electronic communications means comprise an e-mail system.

53. The method of claim 51 wherein the electronic communications means comprise a telephone.

54. The method of claim 51 wherein the electronic communications means comprise a beeper.

55. The method of claim 51 wherein the electronic communications means comprise a personal digital assistant.

56. The method of claim 50 wherein the step of analyzing the relationship records includes:

duplicating the relationship records on at least one secondary database;

distributing received data to the at least one secondary database for analysis based upon a work load criteria; and

issuing the alert meeting the criteria of a user-defined alert rule from the at least one secondary database.

57. The method of claims 1 or 28 wherein the step of utilizing an algorithm to process the received data further comprises transferring the stored processed data to at least one secondary database utilizing the algorithm.

58. The method of claim 57 wherein the step of transferring the stored processed data to at least one secondary database is performed in real-time.

59. The method of claim 57 wherein the step of transferring the stored processed data to at least one secondary database is performed in batch.

60. A method for processing data comprising the steps of:

receiving data comprising at least one record having at least one identifier, each record representing at least one of a plurality of entities;

utilizing an algorithm to;

retrieve from a database a group of additional records having identifiers similar to the identifiers in the received data,

analyze each identifier of the retrieved group of records for a match to at least a portion of the received data,

match the at least a portion of the received data with at least one analyzed record of the retrieved group of records that is determined to reflect a record having identifiers representing an identical one of the plurality of entities,

analyze whether at least one identifier is included in the at least a portion of the received data that was not previously stored in the at least one analyzed record of the retrieved group of records that is determined to reflect a record having identifiers representing an identical one of the plurality of entities; and

re-analyze each identifier of the retrieved group of records for a match to;

the at least a portion of the received data and

the analyzed record of the retrieved group of records that is determined to reflect a record having identifiers representing an identical one of the plurality of entities; and

storing the matched records in the database.

61. The method of claim 60 wherein the step of utilizing an algorithm includes assigning a persistent key.

62. The method of claim 60 wherein the step of utilizing an algorithm further comprises retrieving from the database an additional group of records having identifiers similar to the identifiers in:

at least a portion of the received data and

the analyzed record of the retrieved group of records that is determined to reflect a record having identifiers representing an identical one of the plurality of entities prior to re-analyzing each identifier of the retrieved group of records for a match.

63. The method of claims 60 or 62 wherein the step of utilizing an algorithm includes repeating:

retrieving from a database a group of additional records;

analyzing each identifier of the retrieved group of records;

matching at least a portion of the received data;

analyzing whether at least one identifier is included in the at least a portion of the received data that was not previously stored;

retrieving from the database an additional group of records; and

re-analyzing each identifier of the retrieved group of records for a match until no additional matches are determined.

64. The method of claim 63 wherein the steps of receiving data, utilizing an algorithm and storing the matched records are performed in real-time.

65. The method of claim 63 wherein the steps of receiving data, utilizing an algorithm and storing the matched records are performed in batch.

66. The method of claim 60 wherein the step of utilizing an algorithm includes:

determining whether a particular identifier is one of;

common across records representing at least two different entities and

generally distinctive of a record representing a particular entity; and

separating records that were previously matched based on a particular identifier if the particular identifier is determined to be common across records representing at least two different entities and not generally distinctive of a record representing a particular entity.

67. The method of claim 66 wherein the step of utilizing an algorithm includes prohibiting any additional matches of records based on a particular identifier if the particular identifier is determined to be common across records representing at least two different entities and not generally distinctive of a particular entity.

68. The method of claim 66 wherein the step of utilizing an algorithm includes re-processing the separated records as received data.

69. The method of claim 66 wherein the steps of determining whether a particular identifier is one of common across records representing at least two entities and generally distinctive of a record representing a particular entity and separating records that were previously matched are performed in real-time.

70. The method of claim 66 wherein the steps of determining whether a particular identifier is one of common across records representing at least two different entities and generally distinctive of a record representing a particular entity and separating records that were previously matched are performed in batch.

71. The method of claim 60 wherein the step of utilizing an algorithm includes:

comparing the received data with at least one stored record to determine the existence of a relationship; and

creating a relationship record for each stored record determined to reflect a relationship with at least a portion of the received data.

72. The method of claim 71 wherein the step of utilizing an algorithm includes creating at least one confidence indicator for each relationship record.

73. The method of claim 72 wherein the steps of comparing the received data, creating a relationship record, and creating at least one confidence indicator are performed in real-time.

74. The method of claim 72 wherein the steps of comparing the received data, creating a relationship record, and creating at least one confidence indicator are performed in batch.

75. The method of claim 72 wherein at least one of the confidence indicators indicates the likelihood of a relationship between:

an entity represented by the particular record having a relationship with the portion of the received data, and

an entity represented by the portion of the received data.

76. The method of claim 72 wherein at least one of the confidence indicators indicates the likelihood that:

an entity represented by the particular record having a relationship with the portion of the received data, and

an entity represented by the portion of the receive data are the same.

77. The method of claim 72 wherein the step of utilizing an algorithm includes analyzing the relationship records to determine whether the relationship records reflect at least one relationship not previously determined.

78. The method of claim 77 wherein the step of analyzing the relationship records includes analyzing relationship records reflecting at least one level of degree of separation.

79. The method of claim 78 wherein the step of analyzing relationship records reflecting at least one level of degree of separation includes analyzing relationship records meeting at least one user-defined criterion.

80. The method of claim 79 wherein the step of analyzing relationship records meeting a user-defined criterion includes limiting the relationship records analyzed to a maximum level of degrees of separation.

81. The method of claim 79 wherein the step of analyzing relationship records meeting a user-defined criterion includes limiting the relationship records analyzed to relationship records that include confidence indicators greater than a minimum amount.

82. The method of claim 77 wherein the step of utilizing an algorithm further comprises issuing an alert based upon at least one user-defined alert rule.

83. The method of claim 82 wherein the step of issuing an alert based upon at least one user-defined alert rule includes having the alert communicated via electronic communications means.

84. The method of claim 83 wherein the electronic communications means comprise an e-mail system.

85. The method of claim 83 wherein the electronic communications means comprise a telephone.

86. The method of claim 83 wherein the electronic communications means comprise a beeper.

87. The method of claim 83 wherein the electronic communications means comprise a personal digital assistant.

88. The method of claim 82 wherein the step of analyzing the relationship records includes:

duplicating the relationship records on at least one secondary database;

distributing received data to the at least one secondary for analysis based upon a work load criteria; and

issuing the alert based upon the user-defined alert rule from the at least one secondary database.

89. The method of claim 60 further comprising the step of converting the received data into a standardized message format prior to the step of utilizing an algorithm.

90. The method of claim 60 wherein the step of utilizing an algorithm includes retaining an attribution of each of the identifiers.

91. The method of claim 90 wherein the step of retaining the attribution of each record includes retaining an identity of:

a source system providing each record and

a unique identifier representing each record in the source system.

92. The method of claim 90 wherein the step of retaining an attribution of each of the identifiers includes retaining an identity of a query system and a particular user.

93. The method of claim 60 wherein the step of utilizing an algorithm includes analyzing the received data prior to one of storage in the database and query in the database.

94. The method of claim 93 wherein the step of analyzing the received data prior to one of storage in the database and query in the database includes comparing at least one of the identifiers against one of:

a user-defined criterion and

at least one data set in one of a secondary database and a list.

95. The method of claim 94 wherein the compared identifier is a name of at least one of the plurality of entities and the data set is in a names root list.

96. The method of claim 94 wherein the compared identifier is an address of at least one of the plurality of entities and the data set is in an address list.

97. The method of claim 94 wherein the step of comparing at least one of the identifiers against a user-defined criterion includes formatting at least one identifier in accordance with the user-defined criterion.

98. The method of claim 93 wherein the step of analyzing the received data prior to one of storage in the database and query in a database includes enhancing the received data.

99. The method of claim 98 wherein the step of enhancing the received data includes:

querying at least one data set in one of the secondary database and the list for additional identifiers for the received data; and

supplementing the received data with the additional identifiers.

100. The method of claim 99 wherein the step of querying at least one data set includes:

at least one data set being in the secondary database utilizing the algorithm to query additional databases to locate additional identifiers relating to at least one of the received identifiers; and

supplementing the received data with the additional identifiers located in the secondary database.

101. The method of claim 93 wherein the step of utilizing an algorithm includes creating hash keys of the identifiers.

102. The method of claim 60 wherein the step of utilizing an algorithm includes storing in the database processed queries based upon a user-defined criterion.

103. The method of claim 102 wherein the user-defined criterion includes an expiration date.

104. The method of claim 60 wherein the step of utilizing an algorithm further comprises transferring the stored processed data to at least one secondary database utilizing the algorithm.

105. The method of claim 104 wherein the step of transferring the stored processed data to at least one secondary database is performed in real-time.

106. The method of claim 104 wherein the step of transferring the stored processed data to at least one secondary database is performed in batch.

107. A method for separating previously matched records, the method comprising the steps of:

determining whether a particular identifier in at least one record representing at least one entity is one of;

common across records representing at least two different entities and generally distinctive of a record representing a particular entity; and

separating records that were previously matched based on a particular identifier if the particular identifier is determined to be common across records representing at least two different entities and not generally distinctive of a record representing a particular entity.

108. The method of claim 107 further comprising prohibiting any additional matches of records based on a particular identifier if the particular identifier is determined to be common across records representing a plurality of entities and not generally distinctive of a record representing an entity.

109. The method of claim 107 further comprising the step of re-processing the separated records.

110. The method of claim 107 wherein the steps of determining whether a particular identifier is one of common across records representing at least two different entities and generally distinctive of a record representing a particular entity and separating records that were previously matched are performed in real-time.

111. The method of claim 107 wherein the steps of determining whether a particular identifier is one of common across records representing at least two different entities and generally distinctive of a record representing a particular entity and separating records that were previously matched are performed in batch.

112. A method for processing data in a database, the method comprising the steps of:

receiving data comprising at least one record having at least one identifier, each record representing at least one of a plurality of entities;

comparing the received data with at least one record stored in a database to determine the existence of a relationship in real-time;

creating a relationship record for each record stored in a database determined to reflect a relationship with at least a portion of received data in real-time; and

storing each relationship record in the database.

113. The method of claim 112 further comprising the step of creating at least one confidence indicator for each relationship record in real time.

114. The method of claim 113 wherein at least one confidence indicator indicates the likelihood of a relationship between:

an entity represented by the particular record having a relationship with the portion of the received data, and

an entity represented by the portion of the received data.

115. The method of claim 113 wherein at least one confidence indicator indicates the likelihood that:

an entity represented by the particular record having a relationship with the portion of the received data, and

an entity represented by the portion of the received data are the same.

116. The method of claims 112 or 113 further comprising the step of analyzing the relationship records to determine whether the relationship records reflect at least one relationship not previously determined.

117. The method of claim 116 wherein the step of analyzing the relationship records includes analyzing relationship records reflecting at least one level of degrees of separation.

118. The method of claim 117 wherein the step of analyzing relationship records reflecting at least one level of degrees of separation includes analyzing relationship records meeting at least one user-defined criterion.

119. The method of claim 118 wherein the step of analyzing relationship records meeting at least one user-defined criterion includes limiting the relationship records analyzed to a maximum level of degrees of separation.

120. The method of claim 118 wherein the step of analyzing relationship records meeting at least one user-defined criterion includes limiting the relationship records analyzed to relationship records that include confidence indicators greater than a minimum amount.

121. The method of claim 116 further comprising the step of issuing an alert based upon at least one user-defined alert rule.

122. The method of claim 121 wherein the step of issuing an alert based upon at least one user-defined alert rule includes having the alert communicated via electronic communication means.

123. The method of claim 122 wherein the electronic communication means comprise an e-mail system.

124. The method of claim 122 wherein the electronic communication means comprise a telephone.

125. The method of claim 122 wherein the electronic communication means comprise a beeper.

126. The method of claim 122 wherein the electronic communication means comprise a personal digital assistant.

127. The method of claim 121 further comprising the step of:

duplicating the relationship records on at least one secondary database;

distributing received data to the at least one secondary database for analysis based upon work load criteria; and

issuing the alert meeting the criteria of a user-defined alert rule from the at least one secondary database.

128. For a system for processing data and a computer readable medium containing program instructions for execution by a computer for performing the method comprising the steps of:

receiving data comprising at least one record having at least one identifier, each record representing at least one of a plurality of entities;

utilizing an algorithm to process the received data;

storing the processed data in a database;

receiving data queries for retrieving at least a portion of the data stored in the database; and

utilizing the algorithm to process the queries.

129. The computer readable medium of claim 128 wherein the entities are people.

130. The computer readable medium of claim 128 wherein the entities are personal property.

131. The computer readable medium of claim 130 wherein the personal property are vehicles.

132. The computer readable medium of claim 128 wherein the entities are real property.

133. The computer readable medium of claim 128 wherein the entities are organizations.

134. The computer readable medium of claim 128 wherein the entities are chemical compounds.

135. The computer readable medium of claim 128 wherein the entities are organic compounds.

136. The computer readable medium of claim 128 wherein the entities are proteins.

137. The computer readable medium of claim 128 wherein the entities are biological structures.

138. The computer readable medium of claim 128 wherein the entities are biometric values.

139. The computer readable medium of claim 128 wherein the entities are atomic structures.

140. The computer readable medium of claim 128 wherein the method further comprises the step of converting the received data into a standardized message format prior to utilizing an algorithm to process the received data.

141. The computer readable medium of claim 128 wherein the step of utilizing an algorithm to process the received data includes retaining an attribution of each record.

142. The computer readable medium of claim 141 wherein the step of retaining an attribution of each record includes retaining an identity of:

a source system providing each record and

a unique identifier representing record in the source system.

143. The computer readable medium of claim 141 wherein the step of retaining an attribution of each record includes retaining an identity of a query system and a particular user.

144. The computer readable medium of claim 128 wherein the step of utilizing an algorithm to process the received data includes analyzing the received data prior to one of storage in the database and query in the database.

145. The computer readable medium of claim 144 wherein the step of analyzing the received data prior to one of storage in the database and query in the database includes comparing at least one of the identifiers against one of:

a user-defined criterion and

at least one data set in one of the database and a list.

146. The computer readable medium of claim 145 wherein the compared identifier is a name of at least one of the plurality of entities and the data set is in a names root list.

147. The computer readable medium of claim 145 wherein the compared identifier is an address of at least one of the plurality of entities and the data set is in an address list.

148. The computer readable medium of claim 145 wherein the step of comparing at least one of the identifiers against a user-defined criterion includes formatting at least one identifier in accordance with a user-defined standard.

149. The computer readable medium of claim 144 wherein the step of analyzing the received data prior to one of storage in the database or query in a database includes enhancing the received data.

150. The computer readable medium of claim 149 wherein the step of enhancing the received data includes:

querying at least one data set in one of a database and list for additional identifiers for the received data, and

supplementing the received data with the additional identifiers.

151. The computer readable medium of claim 150 wherein the step of querying at least one data set includes:

at least one data set being in at least one database utilizing the algorithm to query additional databases to locate additional identifiers relating to at least one of the received identifiers; and

supplementing the received data with the additional identifiers located in at least one additional database.

152. The computer readable medium of claim 144 wherein the step of analyzing the received data prior to one of storage in the database and query in the database includes creating hash keys of the identifiers.

153. The computer readable medium of claim 128 wherein the step of utilizing an algorithm to process received data includes storing in the database processed queries based upon a user-defined criterion.

154. The computer readable medium of claim 153 wherein the user-defined criterion includes an expiration date.

155. The computer readable medium of claim 128 wherein the steps of receiving data comprising at least one record having at least one identifier, each record representing at least one of a plurality of entities, utilizing an algorithm to process the received data, and storing the processed data in a database are performed in real-time.

156. The computer readable medium of claim 128 wherein the steps of receiving data comprising at least one record having at least one identifier, each record representing at least one of a plurality of entities, utilizing an algorithm to process the received data, and storing the processed data in a database are performed in batch.

157. The computer readable medium of claims 128 or 144 wherein the step of utilizing an algorithm to process the received data includes:

retrieving from the database a group of additional records having identifiers similar to the identifiers in the received data;

analyzing each identifier of the retrieved group of records for a match to at least a portion of the received data;

matching at least a portion of the received data with at least one analyzed record of the retrieved group of records that is determined to reflect a record having identifiers representing an identical one of the plurality of entities;

analyzing whether at least one identifier is included in the at least a portion of the received data that was not previously stored in the at least one analyzed record of the retrieved group of records that is determined to reflect a record having identifiers representing an identical one of the plurality of entities; and

re-analyzing each identifier of the retrieved group of records for a match to;

at least a portion of the received data and

the analyzed record of the retrieved group of records that is determined to reflect a record having identifiers representing an identical one of the plurality of entities; and

storing the matched records in the database.

158. The computer readable medium of claim 157 wherein matching at least a portion of the received data with at least one analyzed record includes assigning a persistent key.

159. The computer readable medium of claim 157 wherein the step of utilizing an algorithm to process the received data further comprises retrieving from the database an additional group of records having identifiers similar to the identifiers in:

at least a portion of the received data and

the analyzed record of the retrieved group of records that is determined to reflect a record having identifiers representing an identical one of the plurality of entities prior to re-analyzing each identifier of the retrieved group of records for a match.

160. The computer readable medium of claim 159 wherein utilizing an algorithm to process the received data includes repeating:

retrieving from the database a group of records;

analyzing each identifier of the retrieved group of records;

matching at least a portion of the received data;

analyzing whether at least one identifier is included in the at least a portion of the received data that was not previously stored;

retrieving from the database an additional group of records; and

re-analyzing each identifier of the retrieved group of records for a match until no additional matches are determined.

161. The computer readable medium of claim 157 wherein the step of utilizing an algorithm to process the received data includes:

determining whether a particular identifier is one of;

common across records representing at least two different entities and

generally distinctive of a record representing a particular entity; and

separating records that were previously matched based on a particular identifier if the particular identifier is determined to be common across records representing at least two different entities and not generally distinctive of a record representing a particular entity.

162. The computer readable medium of claim 161 wherein the step of utilizing an algorithm to process the received data includes prohibiting any additional matches of records based on a particular identifier if the particular identifier is determined to be common across records representing at least two different entities and not generally distinctive of a record representing a particular entity.

163. The computer readable medium of claim 161 wherein the step of utilizing an algorithm to process the received data includes re-processing the separated records as received data.

164. The computer readable medium of claim 161 wherein the steps of determining whether a particular identifier is one of common across records representing at least two different entities and generally distinctive of a record representing a particular entity and separating records that were previously matched are performed in real-time.

165. The computer readable medium of claim 161 wherein the steps of determining whether a particular identifier is one of common across records representing at least two different entities and generally distinctive of a record representing a particular entity and separating records that were previously matched are performed in batch.

166. The computer readable medium of claim 157 wherein the step of utilizing an algorithm to process the received data includes:

comparing the received data with at least one stored record to determine the existence of a relationship; and

creating a relationship record for each stored record determined to reflect a relationship with at least a portion of the received data.

167. The computer readable medium of claim 166 wherein the step of utilizing an algorithm to process the received data includes creating at least one confidence indicator for each relationship record.

168. The computer readable medium of claim 167 wherein the steps of comparing the received data, creating a relationship record, and creating at least one confidence indicator are performed in real-time.

169. The computer readable medium of claim 167 wherein the steps of comparing the received data, creating a relationship record, and creating at least one confidence indicator are performed in batch.

170. The computer readable medium of claim 167 wherein at least one of the confidence indicators indicates the likelihood of a relationship between:

an entity represented by the particular record having a relationship with the portion of the received data, and

an entity represented by the portion of the received data.

171. The computer readable medium of claim 167 wherein at least one of the confidence indicators indicates the likelihood that:

an entity represented by the particular record having a relationship with the portion of the received data, and

an entity represented by the portion of the received data are the same.

172. The computer readable medium of claim 167 wherein the step of utilizing an algorithm to process received data includes analyzing the relationship records to determine whether the relationship records reflect at least one relationship not previously determined.

173. The computer readable medium of claim 172 wherein the step of analyzing the relationship records includes analyzing relationship records reflecting at least one level of degrees of separation.

174. The computer readable medium of claim 173 the step of analyzing relationship records reflecting at least one level of degrees of separation includes analyzing relationship records meeting a user-defined criterion.

175. The computer readable medium of claim 174 wherein the step of analyzing relationship records meeting a user-defined criterion includes limiting the relationship records analyzed to a maximum level of degrees of separation.

176. The computer readable medium of claim 174 wherein the step of analyzing relationship records meeting a user-defined criterion includes limiting the relationship records analyzed to relationship records that include confidence indicators greater than a minimum amount.

177. The computer readable medium of claim 172 wherein the step of utilizing an algorithm to process received data further comprises issuing an alert based upon at least one user-defined alert rule.

178. The computer readable medium of claim 177 wherein the step of issuing an alert based upon at least one user-defined alert rule includes having the alert communicated via electronic communications means.

179. The computer readable medium of claim 178 wherein the electronic communications means comprise an e-mail system.

180. The computer readable medium of claim 178 wherein the electronic communications means comprise a telephone.

181. The computer readable medium of claim 178 wherein the electronic communications means comprise a beeper.

182. The computer readable medium of claim 178 wherein the electronic communications means comprise a personal digital assistant.

183. The computer readable medium of claim 177 wherein the step of analyzing the relationship records includes:

duplicating the relationship records on at least one secondary database;

distributing received data to the at least one secondary database for analysis based upon a work load criteria; and

issuing the alert meeting the criteria of a user-defined alert rule from the at least one secondary database.

184. The computer readable medium of claims 128 or 155 wherein the step of utilizing an algorithm to process the received data further comprises transferring the stored processed data to at least one secondary database utilizing the algorithm.

185. The computer readable medium of claim 184 wherein the step of transferring the stored processed data to at least one secondary database is performed in real-time.

186. The computer readable medium of claim 184 wherein the step of transferring the stored processed data to at least one secondary database is performed in batch.

187. For a system for processing data into and in a database and a computer readable medium containing program instructions for execution by a computer for performing the method comprising the steps of:

receiving data comprising at least one record having at least one identifier, each record representing at least one of a plurality of entities;

utilizing an algorithm to;

retrieve from a database a group of additional records having identifiers similar to the identifiers in the received data,

analyze each identifier of the retrieved group of records for a match to at least a portion of the received data,

match at least a portion of the received data with at least one analyzed record of the retrieved group of records that is determined to reflect a record having identifiers representing an identical one of the plurality of entities,

analyze whether at least one identifier is included in the at least a portion of the received data that was not previously stored in the at least one analyzed record of the retrieved group of records that is determined to reflect a record having identifiers representing an identical one of the plurality of entities; and

re-analyze each identifier of the retrieved group of records for a match to;

at least a portion of the received data and

the analyzed record of the retrieved group of records that is determined to reflect a record having identifiers representing an identical one of the plurality of entities; and

storing the matched records in the database.

188. The computer readable medium of claim 187 wherein the step of utilizing an algorithm to the received data with at least one analyzed record includes assigning a persistent key.

189. The computer readable medium of claim 187 wherein the step of utilizing an algorithm further comprises retrieving from the database an additional group of records having identifiers similar to the identifiers in:

at least a portion of the received data and

the analyzed record of the retrieved group of records that is determined to reflect a record having identifiers representing an identical one of the plurality of entities prior to re-analyzing each identifier of the retrieved group of records for a match.

190. The computer readable medium of claims 187 or 189 wherein the step of utilizing an algorithm includes repeating:

retrieving from a database a group of additional records;

analyzing each identifier of the retrieved group of records;

matching at least a portion of the received data;

analyzing whether at least one identifier is included in the at least a portion of the received data that was not previously stored;

retrieving from the database an additional group of records; and

re-analyzing each identifier of the retrieved group of records for a match until no additional matches are determined.

191. The computer readable medium of claim 190 wherein the steps of receiving data, utilizing an algorithm and storing the matched records are performed in real-time.

192. The computer readable medium of claim 190 wherein the steps of receiving data, utilizing an algorithm and storing the matched records are performed in batch.

193. The computer readable medium of claim 187 wherein the step of utilizing an algorithm includes:

determining whether a particular identifier is one of;

common across records representing at least two different entities and

generally distinctive of a record representing a particular entity; and

separating records that were previously matched based on a particular identifier if the particular identifier is determined to be common across records representing at least two different entities and not generally distinctive of a record representing a particular entity.

194. The computer readable medium of claim 193 wherein the step of utilizing an algorithm includes prohibiting any additional matches of records based an a particular identifier if the particular identifier is determined to be common across records representing at least two different entities and not generally distinctive of a particular entity.

195. The computer readable medium of claim 193 wherein the step of utilizing an algorithm includes re-processing the separated records as received data.

196. The computer readable medium of claim 193 wherein the steps of determining whether a particular identifier is one of common across records representing at least two entities and generally distinctive of a record representing a particular entity and separating records that were previously matched are performed in real-time.

197. The computer readable medium of claim 193 wherein the steps of determining whether a particular identifier is one of common across records representing at least two entities and generally distinctive of a record representing a particular entity and separating records that were previously matched are performed in batch.

198. The computer readable medium of claim 187 wherein the step of utilizing an algorithm includes:

comparing the received data with at least one stored record to determine the existence of a relationship; and

creating a relationship record for each stored record determined to reflect a relationship with at least a portion of the received data.

199. The computer readable medium of claim 198 wherein the step of utilizing an algorithm includes creating at least one confidence indicator for each relationship record.

200. The computer readable medium of claim 199 wherein the steps of comparing the received data, creating a relationship record, and creating at least one confidence indicator are performed in real-time.

201. The computer readable medium of claim 199 wherein the steps of comparing the received data, creating a relationship record, and creating at least one confidence indicator are performed in batch.

202. The computer readable medium of claim 199 wherein at least one of the confidence indicators indicates the likelihood of a relationship between:

an entity represented by the particular record having a relationship with the portion of the received data, and

an entity represented by the portion of the received data.

203. The computer readable medium of claim 199 wherein at least one of the confidence indicators indicates the likelihood that:

an entity represented by the particular record having a relationship with the portion of the received data, and

an entity represented by the portion of the receive data are the same.

204. The computer readable medium of claim 199 wherein the step of utilizing an algorithm includes analyzing the relationship records to determine whether the relationship records reflect at least one relationship not previously determined.

205. The computer readable medium of claim 204 wherein the step of analyzing the relationship records includes analyzing relationship records reflecting at least one level of degree of separation.

206. The computer readable medium of claim 205 wherein the step of analyzing relationship records reflecting at least one level of degree of separation includes analyzing relationship records meeting at least one user-defined criterion.

207. The computer readable medium of claim 206 wherein the step of analyzing relationship records meeting a user-defined criterion includes limiting the relationship records analyzed to a maximum level of degrees of separation.

208. The computer readable medium of claim 206 wherein the step of analyzing relationship records meeting a user-defined criterion includes limiting the relationship records analyzed to relationship records that include confidence indicators greater than a minimum amount.

209. The computer readable medium of claim 204 wherein the step of utilizing an algorithm further comprises issuing an alert based upon at least one user-defined alert rule.

210. The computer readable medium of claim 209 wherein the step of issuing an alert based upon at least one user-defined alert rule includes having the alert communicated via electronic communications means.

211. The computer readable medium of claim 210 wherein the electronic communications means comprise an e-mail system.

212. The computer readable medium of claim 210 wherein the electronic communications means comprise a telephone.

213. The computer readable medium of claim 210 wherein the electronic communications means comprise a beeper.

214. The computer readable medium of claim 210 wherein the electronic communications means comprise a personal digital assistant.

215. The computer readable medium of claim 209 wherein the step of analyzing the relationship records includes:

duplicating the relationship records on at least one secondary database;

distributing received data to the at least one secondary for analysis based upon a work load criteria; and

issuing the alert based upon the user-defined alert rule from the at least one secondary database.

216. The computer readable medium of claim 187 further comprising the step of converting the received data into a standardized message format prior to utilizing an algorithm.

217. The computer readable medium of claim 187 wherein the step of utilizing an algorithm includes retaining an attribution of each of the identifiers.

218. The computer readable medium of claim 217 wherein the step of retaining the attribution of each record includes retaining an identity of:

a source system providing each record and

a unique identifier representing each record in the source system.

219. The computer readable medium of claim 217 wherein the step of retaining the attribution of each record includes retaining an identity of a query system and a particular user.

220. The computer readable medium of claim 187 wherein the step of utilizing an algorithm includes analyzing the received data prior to one of storage in a database and query in the database.

221. The computer readable medium of claim 220 wherein the step of analyzing the received data prior to one of storage in the database and query in the database includes comparing at least one of the identifiers against one of:

a user-defined criterion and

at least one data set in one of a database and list.

222. The computer readable medium of claim 221 wherein the compared identifier is a name of at least one of the plurality of entities and the data set is in a names root list.

223. The computer readable medium of claim 221 wherein the compared identifier is an address of at least one of the plurality of entities and the data set is in an address list.

224. The computer readable medium of claim 221 wherein the step of comparing at least one of the identifier against a user-defined criterion includes formatting at least one identifier in accordance with the user-defined criterion.

225. The computer readable medium of claim 220 wherein the step of analyzing the received data prior to one of storage in the database and query in a database includes enhancing the received data.

226. The computer readable medium of claim 225 wherein the step of enhancing the received data includes:

querying at least one data set in one of a database and list for additional identifiers for the received data; and

supplementing the received data with the additional identifiers.

227. The computer readable medium of claim 226 wherein the step of querying at least one data set includes:

at least one data set being in at least one database utilizing the algorithm to query additional databases to locate additional identifiers relating to at least one of the received identifiers; and

supplementing the received data with the additional identifiers located in at least one additional database.

228. The computer readable medium of claim 220 wherein the step of utilizing an algorithm includes creating hash keys of the identifiers.

229. The computer readable medium of claim 187 wherein the step of utilizing an algorithm includes storing in the database processed queries based upon a user-defined criterion.

230. The computer readable medium of claim 229 wherein the user-defined criterion includes an expiration date.

231. The computer readable medium of claim 187 wherein the step of utilizing an algorithm further comprises transferring the stored processed data to at least one secondary database utilizing the algorithm.

232. The computer readable medium of claim 231 wherein the step of transferring the stored processed data to at least one secondary database is performed in real-time.

233. The computer readable medium of claim 231 wherein the step of transferring the stored processed data to at least one secondary database is performed in batch.

234. For a system for separating previously matched records, a computer readable medium containing program instructions for execution by a computer for performing the method comprising the steps of:

determining whether a particular identifier in at least one record representing at least one entity is one of;

common across records representing at least two different entities and

generally distinctive of a record representing a particular entity; and

separating records that were previously matched based on a particular identifier if the particular identifier is determined to be common across records representing at least two different entities and not generally distinctive of a record representing a particular entity.

235. The computer readable medium of claim 234 further comprising prohibiting any additional matches of records based on a particular identifier if the particular identifier is determined to be common across records representing a plurality of entities and not generally distinctive of a record representing an entity.

236. The computer readable medium of claim 234 further comprising the step of reprocessing the separated records.

237. The computer readable medium of claim 234 wherein the steps of determining whether a particular identifier is one of common across records representing at least two different entities and generally distinctive of a record representing a particular entity and separating records that were previously matched are performed in real-time.

238. The computer readable medium of claim 234 wherein the steps of determining whether a particular identifier is one of common across records representing at least two different entities and generally distinctive of a record representing a particular entity and separating records that were previously matched are performed in batch.

239. For a system for processing data in a database, a computer readable medium containing program instructions for execution by a computer for performing the method comprising the steps of:

receiving data comprising at least one record having at least one identifier, each record representing at least one of a plurality of entities;

comparing the received data with at least one record stored in a database to determine the existence of a relationship in real-time;

creating a relationship record for each record stored in a database determined to reflect a relationship with at least a portion of received data in real-time; and

storing each relationship record in the database.

240. The computer readable medium of claim 239 further comprising the step of creating at least one confidence indicator for each relationship record in real time.

241. The computer readable medium of claim 240 wherein at least one confidence indicator indicates the likelihood of a relationship between:

an entity represented by the particular record having a relationship with the portion of the received data, and

an entity represented by the portion of the received data.

242. The computer readable medium of claim 240 wherein at least one confidence indicator indicates the likelihood that:

an entity represented by the particular record having a relationship with the portion of the received data, and

an entity represented by the portion of the received data are the same.

243. The computer readable medium of claims 239 or 240 further comprising the step of analyzing the relationship records to determine whether the relationship records reflect at least one relationship not previously determined.

244. The computer readable medium of claim 243 wherein the step of analyzing the relationship records includes analyzing relationship records reflecting at least one level of degrees of separation.

245. The computer readable medium of claim 244 wherein the step of analyzing relationship records reflecting at least one level of degrees of separation includes analyzing relationship records meeting at least one user-defined criterion.

246. The computer readable medium of claim 245 wherein the step of analyzing relationship records meeting at least one user-defined criterion includes limiting the relationship records analyzed to a maximum level of degrees of separation.

247. The computer readable medium of claim 245 wherein the step of analyzing relationship records meeting at least one user-defined criterion includes limiting the relationship records analyzed to relationship records that include confidence indicators greater than a minimum amount.

248. The computer readable medium of claim 243 further comprising the step of issuing an alert based upon at least one user-defined alert rule.

249. The computer readable medium of claim 248 wherein the step of issuing an alert based upon at least one user-defined alert rule includes having the alert communicated via electronic communication means.

250. The computer readable medium of claim 249 wherein the electronic communication means comprise an e-mail system.

251. The computer readable medium of claim 249 wherein the electronic communication means comprise a telephone.

252. The computer readable medium of claim 249 wherein the electronic communication means comprise a beeper.

253. The computer readable medium of claim 249 wherein the electronic communication means comprise a personal digital assistant.

254. The computer readable medium of claim 248 further comprising the step of:

duplicating the relationship records on at least one secondary database;

distributing received data to the at least one secondary database for analysis based upon work load criteria; and

issuing the alert meeting the criteria of a user-defined alert rule from the at least one secondary database.