System of and method for entity representation splitting without the need for human interaction

• US 9,189,505 B2
• Filed: 08/09/2010
• Issued: 11/17/2015
• Est. Priority Date: 08/09/2010
• Status: Active Grant

• Current Assignee: LexisNexis Risk Data Management, Inc. (RELX PLC)
• Original Assignee: LexisNexis Risk Data Management, Inc. (RELX PLC)
• Inventors: Bayliss, David Alan
• Primary Examiner(s): Alam, Hosain
• Assistant Examiner(s): WITZENBURG, BRUCE A
• Application Number:
  US12/852,612
• Publication Number:
  US 20120036112A1
• Time in Patent Office: 1,926 Days
• Field of Search: 707/769, 707/803
• US Class Current: 1/1
• CPC Class Codes: G06F 16/215 Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
• CPC Class Codes: G06F 16/24578 using ranking

Abstract

Disclosed is a system for, and method of, determining whether records and entity representations should be delinked. The system and method need no human interaction in order to calculate parameters and utilizing formulas used for the delinking decisions.

Claims

1. A computer-implemented process for delinking, based on a bloat index formula, entity representations in an electronic database associated with a population of individuals, the electronic database stored at least partially in a memory and comprising a plurality of entity representations, each entity representation comprising a plurality of linked electronic records that likely refer to a same individual of the population of individuals, each electronic record comprising a plurality of fields, each field capable of containing a field value, the process comprising:

calculating a field inconsistency weight for a plurality of fields in the electronic database, wherein each field inconsistency weight is derived from a field inconsistency probability associated with the corresponding field and each field inconsistency probability reflects a likelihood that an arbitrary entity representation in the electronic database includes records with different field values in the corresponding field;

selecting an entity representation in the electronic database;

calculating, for the selected entity representation, a bloat index reflecting a sum of field inconsistency weights over a plurality of fields common to a plurality of linked electronic records of the selected entity representation;

responsive to a field or record being added to the electronic database, determining, based on the bloat index and a known or expected size of the population of individuals associated with the electronic database, whether there is a sufficiently high confidence level that the plurality of linked electronic records of the selected entity representation do not correspond to the respective same individual; and

delinking, by the processor, in the electronic database, each of the plurality of linked electronic records of the selected entity representation based on the determining;

wherein an individual is at least one of a natural person and company.

2. The process of claim 1, wherein the determining comprises comparing the bloat index to a quantity derived from a threshold probability and the known or expected size of the population of individuals associated with the electronic database.

3. The process of claim 2, wherein the threshold probability, T, is calculated based at least on:

T=log(N)−

log(1−

P)−

1,

wherein N represents the known or expected size of the population of individuals associated with the electronic database and P is a number selected from the interval [0,1).

4. The process of claim 1, wherein each field inconsistency probability, p_f, associated with the respective corresponding field, f, is calculated based at least on:

5. The process of claim 4, wherein K_f represents a number of entity representations including two records with different field values in field f.

6. The process of claim 1, wherein the bloat index, B(C), for the selected entity representation, C, is calculated based at least on:

7. The process of claim 1, wherein each respective field inconsistency weight for fields of the selected entity representation that have different field values in at least two records of the selected entity representation is multiplied by the number of different field values in the corresponding field of the selected entity representation.

8. The process of claim 1, wherein:

prior to the delinking, the selected entity representation is associated with a first Definitive Identifier (DID) and each record of the identified entity representation refers to the first DID, and

after the delinking, each record formerly of the identified entity representation refers to a different DID.

9. A system for delinking, based on a bloat index formula, entity representations in an electronic database representing a population of individuals, the electronic database stored at least partially in a memory and comprising a plurality of entity representations, each entity representation comprising a plurality of linked electronic records that likely refer to a same individual of the population of individuals, each electronic record comprising a plurality of fields, each field capable of containing a field value, the system comprising:

a processor;

a memory operatively coupled to the at least one processor and configured for storing data and instructions that, when executed by the processor, cause the system to perform a process comprising;

calculating a field inconsistency weight for a plurality of fields in the electronic database, wherein each field inconsistency weight is derived from a field inconsistency probability associated with the corresponding field and each field inconsistency probability reflects a likelihood that an arbitrary entity representation includes records with different field values in the corresponding field;

selecting an entity representation in the electronic database;

calculating, for the selected entity representation, a bloat index reflecting a sum of field inconsistency weights over a plurality of fields common to a plurality of linked electronic records of the selected entity representation;

responsive to a field or record being added to the electronic database, determining based on the bloat index and a known or expected number of individuals in the electronic database, whether there is a sufficiently high confidence level that the plurality of linked electronic records of the selected entity representation do not correspond to the respective same individual;

delinking by the processor, in the electronic database, each of the plurality of linked electronic records of the selected entity representation based on the determining;

wherein an individual is at least one of a natural person and a company.

10. The system of claim 9, wherein the determining includes comparing the bloat index to a quantity derived from a threshold probability and the total number of entity representations in the electronic database.

11. The system of claim 10, wherein the threshold probability, T, is calculated based at least on:

T=log(N)−

log(1−

P)−

1,

wherein N represents the total number of entity representations in the electronic database and P is a number greater than or equal to 0 and less than 1.

12. The system of claim 9, wherein each field inconsistency probability, p_f, associated with the respective corresponding field, f, is calculated based at least on:

13. The system of claim 9, wherein the bloat index, B(C), for the selected entity representation, C, is calculated based at least on:

14. The system of claim 9, wherein each respective field inconsistency weight for fields of the selected entity representation that have different field values in at least two records of the selected entity representation is multiplied by the number of different field values in the corresponding field of the selected entity representation.

15. The system of claim 9, wherein:

prior to the delinking, the selected entity representation is associated with a first Definitive Identifier (DID) and each record of the identified entity representation refers to the first DID, and

after the delinking, each record formerly of the identified entity representation refers to a different DID.

16. A computer-implemented process for delinking, based on cleave points, entity representations in an electronic database associated with a population of individuals, the electronic database stored at least partially in a memory and comprising a plurality of entity representations, each entity representation comprising a plurality of linked electronic records that likely refer to a same individual of the population of individuals, each electronic record comprising a plurality of fields, each field capable of containing a field value, the process comprising:

calculating a field inconsistency weight for each of a plurality of fields in the electronic database, wherein each field inconsistency weight is derived from a field inconsistency probability associated with the corresponding field and each field inconsistency probability reflects a likelihood that an arbitrary entity representation in the electronic database includes records with different field values in the corresponding field;

selecting a first subset of fields of the plurality of fields, wherein a sum of field inconsistency weights of the selected subset of fields exceeds a threshold wherein the threshold comprises a quantity derived from a threshold probability and the known or expected number of entity representations in the electronic database;

identifying an entity representation in the electronic database having inconsistent field values, between two records of the entity representation, in each field of the selected subset of fields; and

delinking the identified entity representation along a first cleave point between the two records, wherein two separate entity representations are formed from the corresponding delinking.

17. The process of claim 16, wherein each field inconsistency probability, p_f, associated with the corresponding field, f, is calculated at least based on:

18. The process of claim 16, wherein each field inconsistency probability, p_f, associated with the respective corresponding field, f, is calculated based at least on:

19. The process of claim 16, wherein the threshold, T, is calculated based at least on:

T=log(N)−

log(1−

P)−

1,

wherein N represents the total number of entity representations in the electronic database and P is a number greater than or equal to 0 and less than 1.

20. The process of claim 16, wherein the selecting the subset of fields of the plurality of fields is based on a greedy selection of fields from the plurality of fields by magnitude of the corresponding field inconsistency weight.

21. The process of claim 16, further comprising:

delinking the identified entity representation along a second cleave point between another two records of the entity representation, the another two records having inconsistent field values in each field of the selected subset of fields.

22. The process of claim 16, wherein:

prior to the delinking the identified entity representation is associated with a first Definitive Identifier (DID) and each record of the identified entity representation refers to the first DID, and

after the delinking, the two separate entity representations are associated with different DIDs and each record of the two separate entity representations refers to the DID of the corresponding entity representation.

23. A system for delinking, based on cleave points, entity representations in an electronic database associated with a population of individuals, the electronic database stored at least partially in a memory and comprising a plurality of entity representations, each entity representation comprising a plurality of linked electronic records that likely refer to a same individual of the population of individuals, each electronic record comprising a plurality of fields, each field capable of containing a field value, the system comprising:

a processor;

a memory operatively coupled to the at least one processor and configured for storing data and instructions that, when executed by the processor, cause the system to perform a process comprising;

calculating a field inconsistency weight for each of a plurality of fields in the electronic database, wherein each field inconsistency weight is derived from a field inconsistency probability associated with the corresponding field and each field inconsistency probability reflects a likelihood that an arbitrary entity representation in the electronic database includes records with different field values in the corresponding field;

selecting a first subset of fields of the of the plurality of fields, wherein a sum of field inconsistency weights of the selected subset of fields exceeds a threshold, wherein the threshold comprises a quantity derived from a threshold probability and a known or expected size of the population of individuals associated with the electronic database;

identifying an entity representation in in the electronic database having inconsistent field values, between two records of the entity representation, in each field in the selected subset of fields; and

delinking the identified entity representation along a first cleave point between the two records, wherein two separate entity representations are formed from the corresponding delinking.

24. The system of claim 23, wherein each field inconsistency probability, p_f, associated with the respective corresponding field, f, is calculated based at least on:

25. The system of claim 23, wherein the threshold, T, is calculated based at least on:

T=log(N)−

log(1−

P)−

1,

wherein N represents the total number of entity representations in the electronic database and P is a number greater than or equal to 0 and less than 1.

26. The system of claim 23, wherein the selecting the subset of fields of the plurality of fields is based on a greedy selection of fields from the plurality of fields by magnitude of the corresponding field inconsistency weight.

27. The system of claim 23, the process further comprising:

delinking the identified entity representation along a second cleave point between another two records of the entity representation, the another two records having inconsistent field values in each field of the selected subset of fields.

28. The system of claim 23, wherein:

prior to the delinking, the identified entity representation is associated with a first Definitive Identifier (DID) and each record of the identified entity representation refers to the first DID, and

after the delinking, the two separate entity representations are associated with different DIDs and each record of the two separate entity representations refers to the DID of the corresponding entity representation.