Method for record linkage from multiple sources

• US 9,767,127 B2
• Filed: 05/02/2014
• Issued: 09/19/2017
• Est. Priority Date: 05/02/2013
• Status: Active Grant

• Current Assignee: Outseeker Corp.
• Original Assignee: Outseeker Corp.
• Inventors: Feldschuh, Jonathan
• Primary Examiner(s): PHILLIPS, III, ALBERT M
• Application Number:
  US14/249,773
• Publication Number:
  US 20140330845A1
• Time in Patent Office: 1,236 Days
• CPC Class Codes: G06F 16/215 Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors

Abstract

Embodiments of methods for record linkage and comparing attributes are presented herein. Broadly speaking, embodiments of the present invention associate data records using distance measures and weights. More particularly, embodiments of the present invention generate a weight-based comparison of attributes. More specifically, embodiments of the present invention involve deduplication of records in a single database and linkage of records in multiple databases. In some embodiments of the invention, records may be merged into a master database based on the linkage and comparison of attributes. In addition, embodiments of the present invention may calculate a quality measure to be used in comparing attributes and records.

Claims

1. A computer implemented method of association of data records, comprising:

receiving a first specification for a distance measure for each attribute in a subset of attributes of data records in one or more databases where each said distance measure is between zero and one inclusive;

receiving a second specification for a set of non-negative weights for each attribute in said subset of attributes where the sum of said set of weights is equal to one;

receiving a third specification for one or more threshold values;

calculating a pairwise record distance between a plurality of considered records, wherein the pairwise record distance is the weighted sum based on said non-negative weights and each said distance measure for the entire subset of said subset of attributes, and wherein the pairwise record distance falls between zero and one inclusive;

associating a subset of said considered records where said pairwise record distance for said subset of records is within the applicable said threshold value; and

measuring physical distance between record items based on latitude and longitude, and wherein said physical distance is normalized to be compliant.

2. The method of claim 1, further comprising:

merging into a single database said subset of said considered records where said pairwise record distance for said subset of records is within the applicable said threshold value.

3. The method of claim 2, wherein all of the steps are repeated sequentially based on different parameters for said distance measure, said weights, and said thresholds.

4. The method of claim 1, wherein associating said considered records is prevented where said considered records have missing data in their key attributes.

5. A computer implemented method of association of data records, comprising:

receiving a first specification for a distance measure for each attribute in a subset of attributes of data records in one or more databases where each said distance measure is between zero and one inclusive;

receiving a second specification for a set of non-negative weights for each attribute in said subset of attributes where the sum of said set of weights is equal to one;

receiving a third specification for one or more threshold values;

calculating a pairwise record distance between a plurality of considered records, wherein the pairwise record distance is the weighted sum based on said non-negative weights and each said distance measure for the entire subset of said subset of attributes, and wherein the pairwise record distance falls between zero and one inclusive;

associating a subset of said considered records where said pairwise record distance for said subset of records is within the applicable said threshold value; and

standardizing text attributes in a first stage and a second stage, wherein said first stage is standardizing character-level matching considerations, and wherein said second stage is standardizing content of each attribute.

6. A computer implemented method of association of data records, comprising:

receiving a first specification for a distance measure for each attribute in a subset of attributes of data records in one or more databases where each said distance measure is between zero and one inclusive;

receiving a second specification for a set of non-negative weights for each attribute in said subset of attributes where the sum of said set of weights is equal to one;

receiving a third specification for one or more threshold values;

calculating a pairwise record distance between a plurality of considered records, wherein the pairwise record distance is the weighted sum based on said non-negative weights and each said distance measure for the entire subset of said subset of attributes, and wherein the pairwise record distance falls between zero and one inclusive;

associating a subset of said considered records where said pairwise record distance for said subset of records is within the applicable said threshold value; and

wherein said threshold value differs based on how many said records are found similar using key parameters.

7. A computer implemented method of association of data records, comprising:

receiving a first specification for a distance measure for each attribute in a subset of attributes of data records in one or more databases where each said distance measure is between zero and one inclusive;

receiving a second specification for a set of non-negative weights for each attribute in said subset of attributes where the sum of said set of weights is equal to one;

receiving a third specification for one or more threshold values;

calculating a pairwise record distance between a plurality of considered records, wherein the pairwise record distance is the weighted sum based on said non-negative weights and each said distance measure for the entire subset of said subset of attributes, and wherein the pairwise record distance falls between zero and one inclusive;

associating a subset of said considered records where said pairwise record distance for said subset of records is within the applicable said threshold value;

receiving a fourth specification for a database quality measure;

receiving a fifth specification for a record quality measure;

calculating an entry quality measure for each attribute as the product of said database quality measure and said record quality measure;

calculating a final quality measure for each unique attribute value as the sum of said entry qualities for each identical attribute value; and

selecting one of the unique attribute values for merger into a record in a database, wherein the unique attribute value selected is based on a comparison of said final quality measures.

8. The method of claim 7, further comprising:

merging said unique attribute value selected into a record in said database.

9. The method of claim 7, further comprising:

merging all said attribute values along with said final quality values for each said attribute value as attributes into a record in said database.

10. The method of claim 7, wherein said record quality measure includes attribute values from values in a data store that indicate one or more of the following:

a number of reviews of information pertaining to entities upon which said records are based, a number of website page views for one or more pages relating to the entities upon which said records are based, user assessments of information relating to the entities upon which said records are based, a date of the last review of the information pertaining to the entities upon which the records are based, or whether the information pertaining to the entities upon which the records are based has been confirmed as accurate.

11. The method of claim 7, wherein said database quality measure is determined from one or more of the following:

an amount of data in a database, a rate of data creation in a database, or a number of users of a database.

12. The method of claim 7, further comprising:

merging information from source databases according to said final quality while retaining a plurality of candidate values for a plurality of said attributes along with their said final quality measures.

13. A computer implemented method of association of data records, comprising:

receiving a first specification for a distance measure for each attribute in a subset of attributes of data records in one or more databases where each said distance measure is between zero and one inclusive;

receiving a second specification for a set of non-negative weights for each attribute in said subset of attributes where the sum of said set of weights is equal to one;

receiving a third specification for one or more threshold values;

calculating a pairwise record distance between a plurality of considered records, wherein the pairwise record distance is the weighted sum based on said non-negative weights and each said distance measure for the entire subset of said subset of attributes, and wherein the pairwise record distance falls between zero and one inclusive;

associating a subset of said considered records where said pairwise record distance for said subset of records is within the applicable said threshold value;

receiving a fourth specification for a database quality measure;

receiving a fifth specification for one or more attribute quality measures;

calculating an entry quality measure for each attribute as the product of said database quality measure and said attribute quality measure;

calculating a final quality measure for each unique attribute value as the sum of said entry qualities for each identical attribute value;

selecting one of the unique attribute values for merger into a record in a database, wherein the unique attribute value selected is based on a comparison of said final quality measures; and

merging said unique attribute value into a record in a database base.

14. The method of claim 13, wherein said attribute quality measure includes attribute values from values in a data store that indicate one or more of the following:

a number of reviews of information pertaining to entities upon which said records are based, a number of website page views for one or more pages relating to the entities upon which said records are based, user assessments of information relating to the entities upon which said records are based, a date of the last review of the information pertaining to the entities upon which the records are based, or whether the information pertaining to the entities upon which the records are based has been confirmed as accurate.

15. A computer readable non-transitory storage medium comprising instructions executable by a processor for:

receiving a first specification for a distance measure for each attribute in a subset of attributes of data records in one or more databases where each said distance measure is between zero and one inclusive;

receiving a second specification for a set of non-negative weights for each attribute in said subset of attributes where the sum of said set of weights is equal to one;

receiving a third specification for one or more threshold values;

calculating a pairwise record distance between a plurality of considered records, wherein the pairwise record distance is the weighted sum based on said non-negative weights and each said distance measure for the entire subset of said subset of attributes, and wherein the pairwise record distance falls between zero and one inclusive;

associating a subset of said considered records where said pairwise record distance for said subset of records is within the applicable said threshold value;

receiving a fourth specification for a database quality measure;

receiving a fifth specification for a record quality measure;

calculating an entry quality measure for each attribute as the product of said database quality measure and said record quality measure;

calculating a final quality measure for each unique attribute value as the sum of said entry qualities for each identical attribute value; and

selecting one of the unique attribute values for merger into a record in a database, wherein the unique attribute value selected is based on a comparison of said final quality measures.

16. A computer readable non-transitory storage medium comprising instructions executable by a processor for:

receiving a first specification for a distance measure for each attribute in a subset of attributes of data records in one or more databases where each said distance measure is between zero and one inclusive;

receiving a second specification for a set of non-negative weights for each attribute in said subset of attributes where the sum of said set of weights is equal to one;

receiving a third specification for one or more threshold values;

calculating a pairwise record distance between a plurality of considered records, wherein the pairwise record distance is the weighted sum based on said non-negative weights and each said distance measure for the entire subset of said subset of attributes, and wherein the pairwise record distance falls between zero and one inclusive;

associating a subset of said considered records where said pairwise record distance for said subset of records is within the applicable said threshold value;

receiving a fourth specification for a database quality measure;

receiving a fifth specification for one or more attribute quality measures;

calculating an entry quality measure for each attribute as the product of said database quality measure and said attribute quality measure;

calculating a final quality measure for each unique attribute value as the sum of said entry qualities for each identical attribute value;

selecting one of the unique attribute values for merger into a record in a database, wherein the unique attribute value selected is based on a comparison of said final quality measures; and

merging said unique attribute value into a record in a database base.