System and method for automatic weight generation for probabilistic matching

• US 8,321,383 B2
• Filed: 03/19/2010
• Issued: 11/27/2012
• Est. Priority Date: 06/02/2006
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Schumacher, Scott, Ellard, Scott, Adams, Norman S.
• Primary Examiner(s): Vy, Hung T
• Application Number:
  US12/727,975
• Publication Number:
  US 20100175024A1
• Time in Patent Office: 984 Days
• Field of Search: 707/688, 707/736, 707/737, 707/748, 707/749, 706/45, 706/52, 715/810, 703/2, 382/170, 705/33
• US Class Current: 707/688
• CPC Class Codes: G06F 16/24556 Aggregation; Duplicate elimination

Abstract

Embodiments of the invention provide a system and method of automatically generating weights for matching data records. Each field of a record may be compared by an exact match and/or close matches and each comparison can result in a mathematical score which is the sum of the field comparisons. To sum up the field scores accurately, the automatic weight generation process comprises an iterative process. In one embodiment, initial weights are computed based upon unmatched-set probabilities and default discrepancy weights associated with attributes in the comparison algorithm. A bulk cross-match is performed across the records using the initial weights and a candidate matched set is computed for updating the discrepancy probabilities. New weights are computed based upon the unmatched probabilities and the updated discrepancy probabilities. Test for convergence between the new weights and the old weights. Repeat with the new weight table until the weights converge to their final value.

Claims

1. A computer-implemented method of automatically generating weights for associating a plurality of data records from one or more data sources at one or more physical locations, comprising:

a) generating unmatched probabilities for a set of candidate data records, wherein the unmatched probabilities are computed per attribute for each pair of data records in the set of candidate data records;

b) comparing each pair of data records in the set of candidate data records using current weights for selected attributes;

c) determining a candidate matched set with results from the comparing step;

d) generating true discrepancy probabilities with scoring information from the candidate matched set;

e) calculating new weights for the selected attributes based upon the unmatched probabilities and the true discrepancy probabilities to adjust performance of the association of data records; and

repeating steps b)-e) using the new weights if a difference between the current weights and the new weights is larger than a predetermined amount.

2. The method according to claim 1, wherein the set of candidate data records is a subset of the data records from the plurality of data sources.

3. The method according to claim 1, further comprising:

determining default discrepancy probabilities per attribute for each pair of data records in the set of candidate data records based upon a data quality parameter; and

calculating the current weights per attribute based upon the unmatched probabilities and the default discrepancy probabilities.

4. The method according to claim 1, further comprising:

providing a user interface through which a user is able to review and modify a list identifying candidate anonymous data derived from the data records from the plurality of data sources or a subset thereof.

5. The method according to claim 1, further comprising:

storing review results in a non-volatile memory, wherein the review results include information on matched pairs and unmatched pairs in the data records from the plurality of data sources or a subset thereof.

6. The method according to claim 5, further comprising:

repeating steps b)-e) using the stored review results.