COLUMN WEIGHT CALCULATION FOR DATA DEDUPLICATION

• US 20170351717A1
• Filed: 06/02/2016
• Published: 12/07/2017
• Est. Priority Date: 06/02/2016
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Kabra, Namit, Saillet, Yannick
• Granted Patent: US 10,452,627 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 16/1748 De-duplication implemented within the file system, e.g. based on file segments de-duplication techniques in storage systems for the management of data blocks G06F3/0641
• CPC Class Codes: G06F 16/215 Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
• CPC Class Codes: G06F 16/217 Database tuning G06F16/2282 takes precedence; database performance monitoring G06F11/3409

Abstract

A computer system with the capability to identify potentially duplicative records in a data set is provided. A computer may collect a data profile for the data set that provides descriptive information with regard to attributes of the data set. Based, at least in part, on the data profile, weights are determined for the attributes. As values of a data record are compared to values of the same respective attributes in other records, the overall likelihood of a match or duplicate, as indicated by the degree of similarity between values, is modified based on the determined weights associated with the respective attributes.

Claims

1. A computer-implemented method for identifying potentially duplicative records in a data set, the method comprising:

collecting a data profile on the data set, where the data set comprises a plurality of data records organized by distinct attributes, such that separate elements of each record belong to a distinct attribute of the data set, and where the data profile provides descriptive information regarding the attributes, including at least a data classification for one or more of the attributes;

determining a weight to be associated with a particular attribute based, at least in part, on the data profile;

comparing at least one element of a first record of the data set to at least one element of a second record of the data set, where both the element of the first record and the element of the second record are associated with the particular attribute of the data set, to determine a degree of similarity between the two elements; and

determining a likelihood that the first and second records are duplicative over each other, based, at least in part, on the degree of similarity between the two compared elements, where the effect that the degree of similarity between the two compared elements has on the overall determination as to the likelihood that the first and second records are duplicative is increased or decreased based on the determined weight associated with the particular attribute.

2. The computer-implemented method of claim 1, further comprising:

prior to collecting the data profile on the data set, normalizing data elements, across the plurality of data records, associated with the particular attribute.

3. The computer-implemented method of claim 1, further comprising:

subsequent to collecting the data profile on the data set, and prior to determining the weight to be associated with the particular attribute, normalizing data elements, across the plurality of data records, associated with the particular attribute.

4. The computer-implemented method of claim 1, wherein the data profile further includes one or more descriptors for the particular attribute selected from the group consisting of:

a number of distinct data elements associated with the particular attribute, a number of null values for elements associated with the particular attribute, a percentage of null values for elements associated with the particular attribute out of a total number of records, data format for elements associated with the particular attribute, data type for elements associated with the particular attribute, and data quality for elements associated with the particular attribute.

5. The computer-implemented method of claim 1, wherein determining the weight to be associated with the particular attribute comprises:

identifying a data classification from the data profile for the particular attribute; and

based on the identified data classification, determining a corresponding weight.

6. The computer-implemented method of claim 5, further comprising adjusting the weight up or down based on a determination of whether the particular attribute is an optional field or a mandatory field.

7. The computer-implemented method of claim 5, further comprising adjusting the weight up or down based on data quality with respect to elements of the particular attribute.

8. The computer-implemented method of claim 1, further comprising:

determining that all data elements associated with a separate attribute have the same value, and in response, removing the separate attribute from consideration with regard to duplicate record comparisons.

9. The computer-implemented method of claim 1, wherein the likelihood that the first and second records are duplicative over each other is represented by a numerical score, where the numerical score is adjusted based on the degree of similarity between elements of the same attribute, for each compared attribute of the first and second record, as filtered by a respective weight associated with the respective compared attribute.

10. A computer system for identifying potentially duplicative records in a data set, the computer system comprising:

one or more computer processors;

one or more computer-readable storage media;

program instructions, stored on at least one of the one or more computer-readable media for execution by at least one of the one or more computer processors, the program instructions comprising;

program instructions to collect a data profile on the data set, where the data set comprises a plurality of data records organized by distinct attributes, such that separate elements of each record belong to a distinct attribute of the data set, and where the data profile provides descriptive information regarding the attributes, including at least a data classification for one or more of the attributes;

program instructions to determine a weight to be associated with a particular attribute based, at least in part, on the data profile;

program instructions to compare at least one element of a first record of the data set to at least one element of a second record of the data set, where both the element of the first record and the element of the second record are associated with the particular attribute of the data set, to determine a degree of similarity between the two elements; and

program instructions to determine a likelihood that the first and second records are duplicative over each other, based, at least in part, on the degree of similarity between the two compared elements, where the effect that the degree of similarity between the two compared elements has on the overall determination as to the likelihood that the first and second records are duplicative is increased or decreased based on the determined weight associated with the particular attribute.

11. The computer system of claim 10, further comprising program instructions, stored on at least one of the one or more computer-readable media for execution by at least one of the one or more computer processors, to normalize data elements, across the plurality of data records, associated with the particular attribute.

12. The computer system of claim 10, wherein the data profile further includes one or more descriptors for the particular attribute selected from the group consisting of:

a number of distinct data elements associated with the particular attribute, a number of null values for elements associated with the particular attribute, a percentage of null values for elements associated with the particular attribute out of a total number of records, data format for elements associated with the particular attribute, data type for elements associated with the particular attribute, and data quality for elements associated with the particular attribute.

13. The computer system of claim 10, wherein the program instructions to determine the weight to be associated with the particular attribute comprise:

program instructions to identify a data classification from the data profile for the particular attribute; and

program instructions to determine, based on the identified data classification, a corresponding weight.

14. The computer system of claim 13, wherein the program instructions to determine the weight to be associated with the particular attribute further comprise:

program instructions to adjust the corresponding weight up or down based on a determination of whether the particular attribute is an optional field or a mandatory field; and

program instructions to adjust the corresponding weight up or down based on data quality of the data elements associated with the particular attribute.

15. The computer system of claim 10, further comprising program instructions, stored on at least one of the one or more computer-readable media for execution by at least one of the one or more computer processors, to determine that all data elements associated with a separate attribute have the same value, and in response, remove the separate attribute from consideration with regard to duplicate record comparisons.

16. A computer program product for identifying potentially duplicative records in a data set, the computer program product comprising one or more computer-readable storage media, and program instructions, stored on the one or more computer-readable media, the program instructions comprising:

program instructions to collect a data profile on the data set, where the data set comprises a plurality of data records organized by distinct attributes, such that separate elements of each record belong to a distinct attribute of the data set, and where the data profile provides descriptive information regarding the attributes, including at least a data classification for one or more of the attributes;

program instructions to determine a weight to be associated with a particular attribute based, at least in part, on the data profile;

program instructions to compare at least one element of a first record of the data set to at least one element of a second record of the data set, where both the element of the first record and the element of the second record are associated with the particular attribute of the data set, to determine a degree of similarity between the two elements; and

program instructions to determine a likelihood that the first and second records are duplicative over each other, based, at least in part, on the degree of similarity between the two compared elements, where the effect that the degree of similarity between the two compared elements has on the overall determination as to the likelihood that the first and second records are duplicative is increased or decreased based on the determined weight associated with the particular attribute.

17. The computer program product of claim 16, wherein the data profile further includes one or more descriptors for the particular attribute selected from the group consisting of:

a number of distinct data elements associated with the particular attribute, a number of null values for elements associated with the particular attribute, a percentage of null values for elements associated with the particular attribute out of a total number of records, data format for elements associated with the particular attribute, data type for elements associated with the particular attribute, and data quality for elements associated with the particular attribute.

18. The computer program product of claim 16, wherein the program instructions to determine the weight to be associated with the particular attribute comprise:

program instructions to identify a data classification from the data profile for the particular attribute; and

program instructions to determine, based on the identified data classification, a corresponding weight.

19. The computer program product of claim 18, wherein the program instructions to determine the weight to be associated with the particular attribute further comprise:

program instructions to adjust the corresponding weight up or down based on a determination of whether the particular attribute is an optional field or a mandatory field; and

program instructions to adjust the corresponding weight up or down based on data quality of the data elements associated with the particular attribute.

20. The computer program product of claim 16, further comprising program instructions, stored on the one or more computer-readable media, to determine that all data elements associated with a separate attribute have the same value, and in response, remove the separate attribute from consideration with regard to duplicate record comparisons.