Method and system for processing and linking data records

• US 7,912,842 B1
• Filed: 02/04/2003
• Issued: 03/22/2011
• Est. Priority Date: 02/04/2003
• Status: Active Grant

• Current Assignee: LexisNexis Risk Data Management, Inc. (RELX PLC)
• Original Assignee: LexisNexis Risk Data Management, Inc. (RELX PLC)
• Inventors: Bayliss, David
• Primary Examiner(s): Padmanabhan; Kavita
• Application Number:
  US10/357,418
• Time in Patent Office: 2,968 Days
• Field of Search: 707/100, 707/102, 707/103.R, 707/103.Z, 706/45, 706/46, 706/48, 706/52
• US Class Current: 707/749
• CPC Class Codes: G06F 16/215 Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
• CPC Class Codes: G06F 16/2471 Distributed queries

Abstract

Various exemplary systems and methods for linking entity references and identifying associations are presented. In particular, a method is provided for linking a plurality of entity references to at least one entity. The method comprises the steps of evaluating a probability of a match between a first entity reference and a second entity reference based at least in part on a statistical significance of one or more field values being common to both the first entity reference and the second entity reference, wherein field value statistical significance is inversely related to a number of field value occurrences occurring in some or all of the plurality of entity references and linking the first entity reference with the second entity reference when the probability is greater than or equal to a match threshold.

Claims

1. A method for linking a plurality of entity references to at least one entity stored in a computer comprising the steps of:

evaluating, using a computing matrix, a probability of a match between a first entity reference and a second entity reference based at least in part on a statistical significance of a combination of a plurality of field values common to both the first entity reference and the second entity reference, wherein the statistical significance is inversely related to a number of entity references having the combination of plurality of field values from among some or all entity references, wherein the first entity reference is a member of the at least one entity stored in the computer;

determining, during the evaluating of the probability of a match between the first entity reference and the second entity reference, whether a field value of one of the two entity references is a probable synonym of a field value of the other entity reference;

substituting the probable synonym into the field value of the other entity reference and repeating the evaluating of the probability of a match, upon determining that the field value of one of the two entity references is a probable synonym of the field value of the other entity reference; and

linking, using the computing matrix, the first entity reference with the second entity reference when the probability is greater than or equal to a match threshold.

2. The method as in claim 1, wherein the steps of evaluating and linking are repeated for at least one entity reference pairing possible from one or more of the plurality of entity references.

3. The method as in claim 1, further comprising the step of determining probable synonyms among the one or more field values of the entity references, wherein a probability that a first field value is a synonym of a second field value is based at least in part on a frequency of occurrence of both the first field value and the second field value among linked entity references.

4. The method as in claim 1, wherein the step of linking the first entity reference and the second entity reference comprises associating a Definitive Identifier (DID) of a common entity with each of the first entity reference and second entity reference.

5. The method as in claim 1, wherein the probability of a match between the first entity reference and the second entity reference is further based in part on a context of the one or more field values.

6. The method as in claim 5, wherein the context of the one or more field values includes one of a group consisting of:

race;

ethnicity;

geographic location;

geographic proximity;

social proximity;

familial relationships;

gender;

nation of origin;

age;

employment;

education; and

religion.

7. The method as in claim 1, wherein the at least one entity includes one of a group consisting of:

a person;

a business entity;

an organization;

a motor vehicle;

a merchant vessel; and

real property.

8. The method as in claim 1, further comprising the steps of:

evaluating a probability of a match between the second entity reference and a third entity reference based at least in part on a statistical significance of one or more field values common to both the second entity reference and the third entity reference, wherein the statistical significance of a field value is inversely related to a number of occurrences of the field value occurring in some or all of the plurality of entity references; and

linking the second entity reference and the third entity reference when the probability of a match between the second and third entity reference is greater than or equal to the match threshold.

9. The method as in claim 8, further comprising the step of linking the first entity reference to the third entity reference when a field value of a first data field of the first entity reference substantially matches a field value of the first data field of the second entity reference and a field value of a second data field of the second entity reference substantially matches a field value of the second data field of the third entity reference.

10. The method as in claim 9, wherein the steps of linking the first entity reference, second entity reference and third entity reference comprises associating a same Definitive Identifier (DID) of a common entity with the first entity reference, the second entity reference, and the third entity reference.

11. The method as in claim 1, wherein the plurality of entity references are at least part of one or more databases.

12. The method as in claim 11, wherein the one or more databases includes one of a group consisting of:

a medical records database;

a criminal records database;

a motor vehicles registration database;

a voters registration database;

a Uniform Commercial Code filings database;

a corporations filing database;

a Federal Aviations Administration database;

a credit report database;

an immigrations database;

a bankruptcy filings database;

a property deeds database; and

a directory assistance database.

13. A method for linking a plurality of entity references to at least one entity stored in a computer, the entity references comprising a plurality of common data fields, the method comprising the steps of:

determining, using a computing matrix, a number of occurrences of entity references having a combination of a plurality of field values, wherein the plurality of field values comprises a particular field value;

determining, using the computing matrix, a content weight, for the particular field value, wherein the content weight is inversely related to the number of entity references having the combination of the plurality of field values;

determining, using the computing matrix, a match probability between a first entity reference and a second entity reference, wherein the first entity reference is a member of the at least one entity stored in a computer, and wherein the probability is related to the content weight for the particular field value;

determining, during the determining of the match probability between the first entity reference and the second entity reference, whether a field value of one of the two entity references is a probable synonym of a field value of the other entity reference;

substituting the probable synonym into the field value of the other entity reference and repeating the determining of the match probability, upon determining that the field value of one of the two entity references is a probable synonym of the field value of the other entity reference; and

linking, using the computing matrix, the first entity reference and the second entity reference when the match probability is greater than or equal to a match threshold.

14. The method as in claim 13, wherein the content weight is further inversely related to a cautiousness value greater than or equal to zero.

15. The method as in claim 13, wherein the steps of determining a match probability and linking are repeated for at least one entity reference pairing possible one or more of the plurality of entity references.

16. The method as in claim 13, further comprising the step of assigning a link strength to the link between the first entity reference and the second entity reference based in part on the match probability.

17. The method as in claim 13, further comprising the step of replacing a field value with a substitute value in the entity references when the number of occurrences of the field value is greater than a defined threshold.

18. The method as in claim 13, further comprising the steps of:

comparing at least one field value of at least one entity reference with at least one invalid field value; and

replacing the at least one field value with a substitute field value when the at least one field value is substantially equivalent to at least one invalid field value.

19. The method as in claim 13, further comprising the step of linking a third entity with the first entity reference and the second entity reference when a match probability between the second entity reference and the third entity reference is greater than or equal to the match threshold, wherein the match probability between the second entity reference and the third entity reference is related to a sum of the content weights associated with at least one field value common to both the second entity reference and the third entity reference.

20. The method as in claim 13, wherein the match probability between the first entity reference and the second entity reference is further based in part on a context of the at least one field value.

21. The method as in claim 20, wherein the context of the at least one field value includes one of a group consisting of:

race;

ethnicity;

geographic location;

geographic proximity;

social proximity;

familial relationships;

gender;

nation of origin;

age;

employment;

education; and

religion.

22. The method as in claim 13, wherein the at least one entity includes one of a group consisting of:

a person;

a business entity;

an organization;

a motor vehicle;

a merchant vessel; and

real property.

23. The method as in claim 13, wherein the plurality of entity references are at least part of one or more databases.

24. The method as in claim 23, wherein the one or more databases includes one of a group consisting of:

a medical records database;

a criminal records database;

a motor vehicles registration database;

a voters registration database;

a Uniform Commercial Code filings database;

a corporations filing database;

a Federal Aviations Administration database;

a credit report database;

an immigrations database;

a bankruptcy filings database;

a property deeds database; and

a directory assistance database.

25. A method for linking a plurality of entity references to at least one entity stored in a computer comprising the steps of:

determining, using a computing matrix, a match probability between a first entity reference and a second entity reference based at least in part on a statistical significance of a combination of a plurality of data field values common to both the first entity reference and the second entity reference, wherein the statistical significance is inversely related to a number of occurrences of entity references having the combination of plurality of field values from among some or all entity references, wherein the first entity reference is a member of the at least one entity stored in the computer;

determining, during the determining of the match probability between the first entity reference and the second entity reference, whether a field value of one of the two entity references is a probable synonym of a field value of the other entity reference;

substituting the probable synonym into the field value of the other entity reference and repeating the determining of the match probability, upon determining that the field value of one of the two entity references is a probable synonym of the field value of the other entity reference; and

linking, using a computing matrix, the first entity reference with the second entity reference when the match probability is greater than or equal to a match threshold.

26. The method as in claim 25, wherein the step of linking the first entity reference with the second entity reference comprises associating a Definitive Identifier (DID) of a common entity with each of the first entity reference and second entity reference.

27. The method as in claim 26, wherein the match probability is further based in part on a context of the field value.

28. The method as in claim 27, wherein the context of the field value includes one of a group consisting of:

race;

ethnicity;

geographic location;

geographic proximity;

social proximity;

familial relationships;

gender;

nation of origin;

age;

employment;

education; and

religion.

29. The method as in claim 25, wherein the steps of determining and linking are repeated for a plurality of entity reference pairs from the plurality of entity references.

30. The method as in claim 25, wherein the match probability is based at least in part on the statistical significance of multiple field values being common among the first entity reference and the second entity reference.

31. A computer readable medium comprising a set of executable instructions being adapted to manipulate a processor to:

evaluate a probability of a match between a first entity reference and a second entity reference based at least in part on a statistical significance of a combination of a plurality of field values common to both the first entity reference and the second entity reference, wherein the statistical significance is inversely related to a number of entity references having the combination of plurality of field values from among some or all entity references, wherein the first entity reference is a member of at least one entity stored in a computer;

determine, during the evaluating of the probability of a match between the first entity reference and the second entity reference, whether a field value of one of the two entity references is a probable synonym of a field value of the other entity reference;

substitute the probable synonym into the field value of the other entity reference and repeating the evaluating of the probability of a match, upon determining that the field value of one of the two entity references is a probable synonym of the field value of the other entity reference; and

link the first entity reference with the second entity reference when the probability is greater than or equal to a match threshold.

32. The computer readable medium as in claim 31, wherein the instructions adapted to manipulate the processor to link the first entity reference and the second entity reference comprise executable instructions to manipulate the processor to associate a Definitive Identifier (DID) of a common entity with each of the first entity reference and second entity reference.

33. The computer readable medium as in claim 31, further comprising executable instructions adapted to manipulate the processor to:

evaluate a probability of a match between the second entity reference and a third entity reference based at least in part on a statistical significance of a combination of a plurality of field values common to both the second entity reference and the third entity reference, wherein the statistical significance is inversely related to a number of occurrences of the combination of plurality of field values occurring in some or all of the plurality of entity references; and

link the second entity reference and the third entity reference when the probability of a match between the second and third entity reference is greater than or equal to the match threshold.

34. A system for linking a plurality of entity references to at least one entity stored in a computer, the system comprising:

memory;

a processor operably connected to the memory; and

a set of executable instructions stored in the memory and being adapted to manipulate the processor to;

evaluate a probability of a match between a first entity reference and a second entity reference based at least in part on a statistical significance of a combination of a plurality of field values common to both the first entity reference and the second entity reference, wherein the statistical significance is inversely related to a number of occurrences of the combination of plurality of field values occurring in some or all of the plurality of entity references, wherein each of the occurrences comprises an entity reference containing the same combination of a plurality of field values common to both the first entity reference and the second entity reference, and wherein the first entity reference is a member of the at least one entity stored in a computer;

determine, during the evaluating of the probability of a match between the first entity reference and the second entity reference, whether a field value of one of the two entity references is a probable synonym of a field value of the other entity reference;

substitute the probable synonym into the field value of the other entity reference and repeating the evaluating of the probability of a match, upon determining that the field value of one of the two entity references is a probable synonym of the field value of the other entity reference; and

link the first entity reference with the second entity reference when the probability is greater than or equal to a match threshold.

35. The system as in claim 34, wherein the set of executable instructions comprise executable instructions adapted to manipulate the processor to associate a Definitive Identifier (DID) of a common entity with each of the first entity reference and second entity reference.

36. The system as in claim 34, wherein the set of executable instructions comprise executable instructions adapted to manipulate the processor to:

evaluate a probability of a match between the second entity reference and a third entity reference based at least in part on a statistical significance of a combination of a plurality of field values common to both the second entity reference and the third entity reference, wherein the statistical significance is inversely related to a number of occurrences of the combination of plurality of field values occurring in some or all of the plurality of entity references; and

link the second entity reference and the third entity reference when the probability of a match between the second and third entity reference is greater than or equal to the match threshold.