System and method for indexing information about entities from different information sources

• US 5,991,758 A
• Filed: 06/06/1997
• Issued: 11/23/1999
• Est. Priority Date: 06/06/1997
• Status: Expired due to Term

• Current Assignee: International Business Machines Corporation
• Original Assignee: Initiate Systems, Inc. (International Business Machines Corporation)
• Inventors: Ellard, Scott
• Primary Examiner(s): Black, Thomas G.
• Assistant Examiner(s): RONES, CHARLES
• Application Number:
  US08/870,688
• Time in Patent Office: 900 Days
• Field of Search: 702/6, 702/513, 702/5, 709/206, 707/6, 707/2
• US Class Current: 1/1
• CPC Class Codes: G06F 16/2228 Indexing structures
• CPC Class Codes: G06F 16/24558 Binary matching operations
• CPC Class Codes: Y10S 707/99932 Access augmentation or optimizing
• CPC Class Codes: Y10S 707/99936 Pattern matching access

Abstract

A system and method for indexing a data record from an information source into a database, the database containing a plurality of data records, is provided comprising receiving a data record from an information source, the received data record having a predetermined number of fields containing information about a particular entity, standardizing and validating the data in the received data record. A system and method is also provided for retrieving records that refer to an entity characterized by a specific set of data values by comparing a predetermined number of fields within the received data record with a predetermined number of fields within the data records already in the database, selecting data records already in the database as candidates having data within some of the predetermined fields that is identical to the data in the fields of the received data record, and scoring the candidates to determine data records having information about the same entity.

Claims

1. A system for associating a data record from an information source into a database, the database containing a plurality of data records, the system comprising:

means for receiving a data record from an information source, the received data record having a predetermined number of fields containing information about a particular entity;

means for comparing selected fields within the received data record with corresponding fields within the data records already in the database;

means, responsive to comparison, for identifying data records already in the database having data within some of the selected fields that match to the data in the fields of the received data record as possible matching candidates, the identifying means further comprising one or more control databases for identifying errors in the data contained in one or more fields of the received data record in order to correct the data in the received data record and means for matching the corrected data in the received data record with the data records already in the database; and

means for scoring the identified matching candidates using a predetermined scoring criteria which measures a likelihood of a match between the received data record and the data records in the database based on the selected fields to determine if the received data record and a data record in the database contains information about the same entity thereby associating data records about the same entity despite errors contained in the data records.

2. The system of claim 1, wherein said scoring means comprises means for generating a score indicating whether each possible matching candidate contains information about the same entity as the received data record despite errors in the possible matching candidate or the received data record, and means for generating a link between the possible matching candidate with the received data record when the generated score is above a predetermined threshold level.

3. The system of claim 2, wherein the means for generating a link further comprises means for querying the data record database to identify any potential matches between the incoming data record and the data records in the data record database, means for merging the incoming data record with a data record in the data record database when a match is identified, and, when a match is not identified, means for allocating new storage space to the incoming data record and means for adding the incoming data record into the new storage space.

4. The system of claim 2, further comprising means for storing the links between the received data record and the data records already in the database in a separate database from said data records.

5. The system of claim 4 further comprising means for performing another scoring for the received data record and the data records in the database using a different threshold value comprising means for deleting the stored links, means for changing the threshold level and means for regenereating links for a data record when the second score is above the changed threshold level.

6. The system of claim 1, wherein said information source further comprises a plurality of information sources having data records containing fields and wherein the comparison means comprises means for comparing the fields in the data records from the plurality of information sources to fields in data records already in the database in order to associate the data records from the plurality of information sources with the data records in the database despite errors in the data records.

7. The system of claim 1, wherein said one or more databases comprises a rules database for storing rules for automatically determining the associations between data records containing information about the same entity, a links database for storing said associations between the data records about a same entity in the data record database, an exception database for storing an action to be taken when a received data record cannot be processed, an anonymous name database for storing known anonymous names which appear in the data records in the data records database, a canonical name database for storing a relationship between a full given name and a nickname that is in a data record in the data record database, and a threshold database for storing a threshold used for the comparison of the data records.

8. The system of claim 1 further comprising means for standardizing the information in an incoming data record before the incoming data record is compared to the other data records in the database in order to reduce the likelihood of a mismatch.

9. The system of claim 1 further comprising means for incorporating the incoming data record into a data record database.

10. The system of claim 9, wherein the incorporating means comprises means for adding a data record containing information about a new entity into the data record database comprising means for determining if the information in the incoming data record matches information in the data record database, means for allocating storage space in the data records database to the data record containing information about the new entity, and means for storing the data record containing information about the new entity in the allocated storage space.

11. The system of claim 9, wherein said incorporating means comprises means for adding a data record containing information about an entity that already has a data record in the data record database comprising means for determining if there are matching data records in the data records database, means for merging the data records in the data records database with the incoming data record if a match is determined, and means for updating a links database to contain an association between the incoming data record and the data records already in the data record database.

12. The system of claim 9, wherein the incorporating means comprises means for adding a data record containing information about an unknown entity into the data record database.

13. The system of claim 1 further comprising a rules database containing rules for determining a match between an incoming data record and data records in the data record database based on the information in the data records, and means for updating the rules database with additional rules.

14. The system of claim 13, wherein said rule database updating means comprises means for comparing a new rule with a rule already in the rules database and means for synthesizing the data records associated with the new rule with the data records associated with a previous rule in the rules database.

15. A system for associating data records from a plurality of sources containing information about the same entity together despite errors in the information contained in the data records, the system comprising:

means for comparing an incoming data record to a database of data records based on a comparison of selected fields in the incoming data record and in the data records in the database to identify matching data records based on the selected fields; and

means for controlling the comparison means comprising one or more control databases for identifying errors in the data contained in one or more fields of the received data record in order to correct the data in the received data record and means for matching the corrected data in the received data record with the data records already in the database, the one or more control databases comprises a rules database for storing rules for automatically determining the associations between data records containing information about the same entity, a links database for storing said associations between the data records about a same entity in a separate database from the data record database, an exception database for storing an action to be taken when a received data record cannot be processed, an anonymous name database for storing known anonymous names which appear in the data records in the data records database, a canonical name database for storing a relationship between a full given name and a nickname that is in a data record in the data record database, and a threshold database for storing a threshold used for the comparison of the data records.

16. A method for associating a data record from an information source into a database, the database containing a plurality of data records, the method comprising:

receiving a data record from an information source, the received data record having a predetermined number of fields containing information about a particular entity;

comparing selected fields within the received data record with corresponding fields within the data records already in the database;

identifying data records already in the database, based on the comparison, having data within some of the selected fields that match to the data in the fields of the received data record as possible matching candidates, the identifying further comprising identifying errors in the data contained in one or more fields of the received data record using one or more control databases in order to correct the data in the received data record and matching the corrected data in the received data record with the data records already in the database; and

scoring the identified matching candidates using a predetermined scoring criteria which measures a likelihood of a match between the received data record and the data records in the database based on the selected fields to determine if the received data record and a data record in the database contains information about the same entity thereby associating data records about the same entity despite errors contained in the data records.

17. The method of claim 16, wherein said scoring comprises generating a score indicating whether each possible matching candidate contains information about the same entity as the received data record despite errors in the possible matching candidate or the received data record, and generating a link between the possible matching candidate with the received data record when the generated score is above a predetermined threshold level.

18. The method of claim 17, wherein the generating a link further comprises querying the data record database to identify any potential matches between the incoming data record and the data records in the data record database, merging the incoming;

data record with a data record in the data record database when a match is identified, and, when a match is not identified, allocating new storage space to the incoming data record and adding the incoming data record into the new storage space.

19. The method of claim 17, further comprising storing the links between the received data record and the data records already in the database in a separate database from said data records.

20. The method of claim 19 further comprising performing another scoring for the received data record and the data records in the database using a different threshold value comprising deleting the stored links, changing the threshold level and regenereating links for a data record when the second score is above the changed threshold level.

21. The method of claim 16, wherein said information source further comprises a plurality of information sources having data records containing fields and wherein the comparison comprises comparing the fields in the data records from the plurality of information sources to fields in data records already in the database in order to associate the data records from the plurality of information sources with the data records in the database despite errors in the data records.

22. The method of claim 16, wherein said one or more databases comprises a rules database for storing rules for automatically determining the associations between data records containing information about the same entity, a links database for storing said associations between the data records about a same entity in the data record database, an exception database for storing an action to be taken when a received data record cannot be processed, an anonymous name database for storing known anonymous names which appear in the data records in the data records database, a canonical name database for storing a relationship between a full given name and a nickname that is in a data record in the data record database, and a threshold database for storing a threshold used for the comparison of the data records.

23. The method of claim 16 further comprising standardizing the information in an incoming data record before the incoming data record is compared to the other data records in the database in order to reduce the likelihood of a mismatch.

24. The method of claim 16 further comprising incorporating the incoming data record into a data record database.

25. The method of claim 24, wherein the incorporating comprises adding a data record containing information about a new entity into the data record database comprising determining if the information in the incoming data record matches information in the data record database, allocating storage space in the data records database to the data record containing information about the new entity, and storing the data record containing information about the new entity in the allocated storage space.

26. The method of claim 24, wherein said incorporating comprises adding a data record containing information about an entity that already has a data record in the data record database comprising determining if there are matching data records in the data records database, merging the data records in the data records database with the incoming data record if a match is determined, and updating a links database to contain an association between the incoming data record and the data records already in the data record database.

27. The method of claim 24, wherein the incorporating comprises means for adding a data record containing information about an unknown entity into the data record database.

28. The method of claim 16 further comprising using a rules database containing rules for determining a match between an incoming data record and data records in the data record database based on the information in the data records, and updating the rules database with additional rules.

29. The method of claim 28, wherein said rule database updating comprises comparing a new rule with a rule already in the rules database and synthesizing the data records associated with the new rule with the data records associated with a previous rule in the rules database.