Internal linking co-convergence using clustering with no hierarchy

• US 9,043,359 B2
• Filed: 09/17/2013
• Issued: 05/26/2015
• Est. Priority Date: 02/04/2003
• Status: Active Grant

• Current Assignee: LexisNexis Risk Solutions, Inc. (RELX PLC)
• Original Assignee: LexisNexis Risk Solutions, Inc. (RELX PLC)
• Inventors: Bayliss, David Alan
• Primary Examiner(s): LEROUX, ETIENNE PIERRE
• Application Number:
  US14/029,698
• Publication Number:
  US 20140032556A1
• Time in Patent Office: 616 Days
• Field of Search: 707/5, 707/770, 706/47, 709204-206
• US Class Current: 707/770
• CPC Class Codes: G06F 16/215 Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
• CPC Class Codes: G06F 16/2246 Trees, e.g. B+trees
• CPC Class Codes: G06F 16/24 Querying
• CPC Class Codes: G06F 16/245 Query processing
• CPC Class Codes: G06F 16/2455 Query execution
• CPC Class Codes: G06F 16/282 Hierarchical databases, e.g. IMS, LDAP data stores or Lotus Notes
• CPC Class Codes: G06F 16/285 Clustering or classification
• CPC Class Codes: G06F 16/35 Clustering; Classification
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques

Abstract

Certain implementations of the disclosed technology include systems and methods for linking entities in an internal database by utilizing co-convergence and clustering. The method may include clustering database records into a first set of clusters having corresponding first cluster identifications (IDs). The clustering may be based at least in part on determining similarity among corresponding field values. The method may include associating mutually matching database records, by performing at least one matching iteration for each of the database records. The method may include determining similarity among corresponding field values of the database records, re-clustering at least a portion of the database records into a second set of clusters, the re-clustering based at least in part on the associating mutually matching database records and on the determining similarity among corresponding field values of the database records.

Claims

1. A computer-implemented method comprising:

clustering database records into a first set of clusters having corresponding first cluster identifications (IDs), each database record comprising one or more field values, wherein the clustering is based at least in part on determining similarity among corresponding field values of the database records;

associating mutually matching database records, wherein the associating comprises performing at least one matching iteration for each of the database records, wherein the matching iteration is based at least in part on the first cluster IDs;

determining similarity among corresponding field values of the database records;

re-clustering at least a portion of the database records into a second set of clusters having corresponding second cluster IDs, the re-clustering based at least in part on the associating mutually matching database records and on the determining similarity among corresponding field values of the database records; and

outputting database record information, based at least in part on the re-clustering.

2. The method of claim 1, wherein the determining similarity among the corresponding field values of the database records comprises:

assigning an n-degree hyperspace attribute to each database record, where n represents a number of data fields of interest, and wherein the hyperspace attribute corresponding to two database records is correlated with a similarity of the corresponding field values of the two database records;

determining membership of each database record in a plurality of hyperspace clusters based at least in part on the hyperspace attributes;

assigning, to each record, a cluster ID and a match value reflecting a likelihood that the record is a member of a particular hyperspace cluster; and

linking related records based at least in part on the cluster ID and match value.

3. The method of claim 2, further comprising merging database records having hyperspace attribute differences within a predefined criteria to eliminate similar exemplars that are likely to represent a same entity, the merging resulting in a reduced set of database records.

4. The method of claim 3, further comprising:

recalculating the field value weights for the reduced set of database records; and

re-clustering the reduced set of records based at least in part on the recalculated field value weights.

5. The method of claim 2, wherein the determining membership of each database record in the plurality of hyperspace clusters further comprises creating a plurality of nodes at random locations in hyperspace, each node maintaining records in the hyperspace based on the hyperspace attribute for which it is the closest node.

6. The method of claim 1, wherein the associating mutually matching database records further comprises:

determining highest compelling linkages among the database records, the determining comprising;

identifying mutually preferred pairs of records from the database records, each mutually preferred pair of records consisting of a first record and a second record, the first record consisting of a preferred record associated with the second record and the second record consisting of a preferred record associated with the first record, wherein the mutually preferred pairs of records each has a match score that meets pre-specified match criteria;

assigning, for each record from the database records, at least one associated preferred record, wherein a match value assigned to a given record together with its associated preferred record is at least as great as a match value assigned to the record together with any other record in the database records; and

forming and storing a plurality of entity representations in the database, each entity representation of the plurality of entity representations comprising at least one linked pair of mutually preferred records.

7. The method of claim 1, wherein each database record corresponds to an entity representation, each database record comprising a plurality of fields, each field configured to contain a field value, and each field value assigned a field value weight corresponding to a specificity of the field value in relation to all field values in a corresponding field of the records.

8. The method of claim 1, wherein performing the at least one matching iteration comprises linking related database records based at least in part on a determined match value or determined mutually preferred records.

9. A system comprising:

at least one memory for storing data and computer-executable instructions; and

at least one processor configured to access the at least one memory and further configured to execute the computer-executable instructions for;

clustering database records into a first set of clusters having corresponding first cluster identifications (IDs), each database record comprising one or more field values, wherein clustering is based at least in part on a determined similarity among corresponding field values of the database records;

associating mutually matching database records by performing at least one matching iteration for each of the database records, wherein the matching iteration is based at least in part on the first cluster IDs;

determining similarity among corresponding field values of the database records;

re-clustering at least a portion of the database records into a second set of clusters having corresponding second cluster IDs, wherein the re-clustering is based at least in part on the associated mutually matching database records and on the determining the similarity among corresponding field values of the database records; and

outputting database record information, based at least in part on the re-clustering.

10. The system of claim 9, wherein the at least one processor is further configured to execute the computer-executable instructions for determining the similarity among the corresponding field values of the database records by:

assigning an n-degree hyperspace attribute to each database record, where n represents a number of data fields of interest, and wherein the hyperspace attribute corresponding to two database records is correlated with a similarity of the corresponding field values of the two database records;

determining membership of each database record in a plurality of hyperspace clusters based at least in part on the hyperspace attributes;

assigning, to each record, a cluster ID and a match value reflecting a likelihood that the record is a member of a particular hyperspace cluster; and

linking related records based at least in part on the cluster ID and match value.

11. The system of claim 10, wherein the at least one processor is further configured to execute the computer-executable instructions to merge database records having hyperspace attribute differences within a predefined criteria to eliminate similar exemplars that are likely to represent a same entity, and resulting in a reduced set of database records.

12. The system of claim 11, wherein the at least one processor is further configured to execute the computer-executable instructions to:

recalculate the field value weights for the reduced set of database records; and

re-cluster the reduced set of records based at least in part on the recalculated field value weights.

13. The system of claim 10, wherein the determining membership of each database record in the plurality of hyperspace clusters further comprises creating a plurality of nodes at random locations in hyperspace, each node maintaining records in hyperspace based on the hyperspace attribute for which it is the closest node.

14. The system of claim 9, wherein the associating mutually matching database records further comprises:

determining highest compelling linkages among the database records, the determining comprising;

identifying mutually preferred pairs of records from the database records, each mutually preferred pair of records consisting of a first record and a second record, the first record consisting of a preferred record associated with the second record and the second record consisting of a preferred record associated with the first record, wherein the mutually preferred pairs of records each has a match score that meets pre-specified match criteria;

assigning, for each record from the database records, at least one associated preferred record, wherein a match value assigned to a given record together with its associated preferred record is at least as great as a match value assigned to the record together with any other record in the database records; and

forming and storing a plurality of entity representations in the database, each entity representation of the plurality of entity representations comprising at least one linked pair of mutually preferred records.

15. The system of claim 9, wherein each database record corresponds to an entity representation, each database record comprising a plurality of fields, each field configured to contain a field value, and each field value assigned a field value weight corresponding to a specificity of the field value in relation to all field values in a corresponding field of the records.

16. The system of claim 9, wherein performing the at least one matching iteration comprises linking related database records based at least in part on a determined match value or determined mutually preferred records.

17. A non-transitory computer readable media comprising computer-executable instructions that, when executed by one or more processors, cause the one or more processors to perform a method comprising:

clustering database records into a first set of clusters having corresponding first cluster identifications (IDs), each database record comprising one or more field values, wherein the clustering is based at least in part on determining similarity among corresponding field values of the database records;

associating mutually matching database records, wherein the associating comprises performing at least one matching iteration for each of the database records, wherein the matching iteration is based at least in part on the first cluster IDs;

determining similarity among corresponding field values of the database records;

re-clustering at least a portion of the database records into a second set of clusters having corresponding second cluster IDs, the re-clustering based at least in part on the associating mutually matching database records and on the determining similarity among corresponding field values of the database records; and

outputting database record information, based at least in part on the re-clustering.

18. The computer-readable media of claim 17, wherein the determining similarity among the corresponding field values of the database records comprises:

assigning an n-degree hyperspace attribute to each database record, where n represents a number of data fields of interest, and wherein the hyperspace attribute corresponding to two database records is correlated with a similarity of the corresponding field values of the two database records;

determining membership of each database record in a plurality of hyperspace clusters based at least in part on the hyperspace attributes;

assigning, to each record, a cluster ID and a match value reflecting a likelihood that the record is a member of a particular hyperspace cluster; and

linking related records based at least in part on the cluster ID and match value.

19. The computer-readable media of claim 18, further comprising merging database records having hyperspace attribute differences within a predefined criteria to eliminate similar exemplars that are likely to represent a same entity, the merging resulting in a reduced set of database records.

20. The computer-readable media of claim 19, further comprising:

recalculating the field value weights for the reduced set of database records; and

re-clustering the reduced set of records based at least in part on the recalculated field value weights.

21. The computer-readable media of claim 18, wherein the determining membership of each database record in the plurality of hyperspace clusters further comprises creating a plurality of nodes at random locations in hyperspace, each node maintaining records in hyperspace based on the hyperspace attribute for which it is the closest node.

22. The computer-readable media of claim 17, wherein the associating mutually matching database records further comprises:

determining highest compelling linkages among the database records, the determining comprising;

identifying mutually preferred pairs of records from the database records, each mutually preferred pair of records consisting of a first record and a second record, the first record consisting of a preferred record associated with the second record and the second record consisting of a preferred record associated with the first record, wherein the mutually preferred pairs of records each has a match score that meets pre-specified match criteria;

assigning, for each record from the database records, at least one associated preferred record, wherein a match value assigned to a given record together with its associated preferred record is at least as great as a match value assigned to the record together with any other record in the database records; and

forming and storing a plurality of entity representations in the database, each entity representation of the plurality of entity representations comprising at least one linked pair of mutually preferred records.

23. The computer-readable media of claim 17, wherein each database record corresponds to an entity representation, each database record comprising a plurality of fields, each field configured to contain a field value, and each field value assigned a field value weight corresponding to a specificity of the field value in relation to all field values in a corresponding field of the records.

24. The computer-readable media of claim 17, wherein performing the at least one matching iteration comprises linking related database records based at least in part on a determined match value or determined mutually preferred records.