Enterprise data duplication identification

• US 8,429,137 B2
• Filed: 09/02/2010
• Issued: 04/23/2013
• Est. Priority Date: 09/02/2010
• Status: Active Grant

• Current Assignee: Federal Express Corporation (FedEx Corporation)
• Original Assignee: Federal Express Corporation (FedEx Corporation)
• Inventors: Brittain, Brian, Cooper, Mark, Yerramsetty, Viswanath
• Primary Examiner(s): Corrielus, Jean M
• Application Number:
  US12/874,391
• Publication Number:
  US 20120059827A1
• Time in Patent Office: 964 Days
• Field of Search: 707/741, 707/754, 707/758, 707/950, 707/692, 707/697, 707/698, 707/699, 707/613, 707/792, 707/705, 707/776
• US Class Current: 707/692
• CPC Class Codes: G06F 16/215 Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
• CPC Class Codes: G06F 16/24556 Aggregation; Duplicate elimination

Abstract

Systems, methods, and computer program products are provided for identifying duplicate data. In one exemplary embodiment, there is provided a method for identifying duplicate data. The method may include identifying one or more reference fields that include one or more data values. The method may include retrieving the one or more reference fields and one or more data values. The method may also include transforming the one or more reference fields into one or more reference fingerprint patterns. The method may also include identifying one or more target fields that include one or more target field values. The method may also include retrieving the one or more target fields. The method may also include transforming the one or more target field values into one or more target fingerprint patterns. The method may also include comparing the one or more reference fingerprint patterns with the one or more target fingerprint patterns. The method may further include determining an overlap between the one or more reference fingerprint patterns and the one or more target fingerprint patterns.

Claims

1. A computer-implemented method for identifying duplicate data, the method comprising the steps, performed by a computer, of:

identifying one or more reference fields that include one or more data values;

retrieving the reference fields;

generating one or more reference fingerprint patterns;

transforming the reference fields into the one or more reference fingerprint patterns;

identifying one or more target fields that include one or more data values;

retrieving the target fields;

generating one or more target fingerprint patterns;

transforming the target fields into the one or more target fingerprint patterns;

comparing the one or more target fingerprint patterns with the one or more reference fingerprint patterns; and

determining an overlap between the one or more target fingerprint patterns and the one or more reference fingerprint patterns to identify duplicate data, wherein the one or more reference fingerprint patterns and one or more target fingerprint patterns include one or more letters and one or more numbers.

2. The method of claim 1, further comprising:

storing the retrieved reference fields and target fields.

3. The method of claim 1, further comprising:

determining at least one of a sample match score, target match, and match probability.

4. The method of claim 3, wherein the match probability is ≧

0.10.

5. The method of claim 3, further comprising:

determining a final score.

6. A computer-readable medium containing instructions which when executed on a processor performs a method for identifying duplicate data, the method comprising:

identifying one or more reference fields that include one or more data values;

retrieving the reference fields;

generating one or more reference fingerprint patterns;

transforming the reference fields into the one or more reference fingerprint patterns;

identifying one or more target fields that include one or more data values;

retrieving the target fields;

generating one or more target fingerprint patterns;

transforming the target fields into the one or more target fingerprint patterns;

comparing the one or more target fingerprint patterns with the one or more reference fingerprint patterns; and

determining an overlap between the one or more target fingerprint patterns and the one or more reference fingerprint patterns to identify duplicate data, wherein the one or more reference fingerprint patterns and one or more target fingerprint patterns include one or more letters and one or more numbers.

7. The computer-readable medium of claim 6, further comprising:

storing the retrieved reference fields and target fields.

8. The computer-readable medium of claim 6, further comprising:

determining at least one of a sample match score, target match, and match probability.

9. The computer-readable medium of claim 8, wherein the match probability is ≧

0.10.

10. The computer-readable medium of claim 8, further comprising:

determining a final score.

11. A system for identifying duplicate data, comprising:

one or more systems that include one or more target fields and data values; and

a duplication identification system comprising one or more processors, wherein the duplication identification system;

identifies one or more reference fields that include one or more data values;

retrieves the reference fields;

generates one or more reference fingerprint patterns;

transforms the reference fields into the one or more reference fingerprint patterns;

identifies one or more target fields that include one or more data values;

retrieves the target fields;

generates one or more target fingerprint patterns;

transforms the target fields into the one or more target fingerprint patterns;

compares the one or more target fingerprint patterns with the one or more reference fingerprint patterns; and

determines an overlap between the one or more target fingerprint patterns and the one or more reference fingerprint patterns to identify duplicate data, wherein the one or more reference fingerprint patterns and one or more target fingerprint patterns include one or more letters and one or more numbers.

12. The system of claim 11, wherein the duplication identification system stores the retrieved reference fields and target fields.

13. The system of claim 11, wherein the duplication identification system determines at least one of a sample match score, target match, and match probability.

14. The system of claim 13, wherein the match probability is ≧

0.10.

15. The system of claim 13, wherein the duplication identification system determines a final score.