Probabilistic model for record linkage

• US 20060179050A1
• Filed: 10/21/2005
• Published: 08/10/2006
• Est. Priority Date: 10/22/2004
• Status: Abandoned Application

• Current Assignee: Siemens Medical Solutions USA Incorporated (Siemens AG)
• Original Assignee: Siemens Medical Solutions USA Incorporated (Siemens AG)
• Inventors: Sandilya, Sathyakama, Rao, R. Bharat, Landi, William A., Giang, Phan H.
• Application Number:
  US11/255,660
• Publication Number:
  US 20060179050A1
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/3346 using probabilistic model

Abstract

A method for probabilistic record linkage includes providing a record pair comprising a plurality of fields, providing a plurality of scenarios, each scenario relating to a distribution of patterns among a plurality of attribute statuses, and comparing the record pair to determine a record difference. The method includes determining a probability of a status for each of a plurality of attributes based on the distance metric of the plurality of fields, wherein each field corresponds to a respective attribute, wherein the field is observable and the attribute is hidden, determining a probability of each scenario based on the probability of the status for each attribute and the Bayesian net representing the probabilistic model on the relationship between scenarios and attributes, and outputting a probability of duplication or non-duplication of the record pair determined from the probabilities of the plurality of scenarios.

Claims

1. A computer-implemented method for probabilistic record linkage comprising:

providing a record pair comprising a plurality of fields;

providing a plurality of scenarios, each scenario relating to a distribution of patterns among a plurality of attribute statuses;

comparing the record pair to determine a record difference;

determining a probability of a status for each of a plurality of attributes based on the distance metric of the plurality of fields, wherein each field corresponds to a respective attribute, wherein the field is observable and the attribute is hidden;

determining a probability of each scenario based on the probability of the status for each attribute and the Bayesian net representing the probabilistic model on the relationship between scenarios and attributes; and

outputting a probability of duplication or non-duplication of the record pair determined from the probabilities of the plurality of scenarios.

2. The computer-implemented method of claim 1, wherein comparing the record pair comprises comparing record values of the record pair field-wise or across fields.

3. The computer-implemented method of claim 1, wherein determining the probability of a status for each of the plurality of attributes comprises:

providing a predefined error rate of data entering in a field;

determining a distance metric between field values; and

determining a probability of making i errors when entering m characters with the predefined error rate.

4. The computer-implemented method of claim 1, wherein each among a plurality of scenarios is characterized by a probability model on patterns of attribute statuses for example Bayesian net, conditional probabilities of attribute status given scenarios.

5. The computer-implemented method of claim 1,

wherein the probability of duplication is compared to a threshold, wherein the threshold corresponds to a significant probability of duplication.

6. The computer-implemented method of claim 1, further comprising:

providing a graphical user interface; and

displaying at least one of a scenario probability, a most probable scenario, a probability of duplication, and/or a probability that an entity is intended by an input search criteria.

7. The computer-implemented method of claim 1, wherein the record pair is a search criteria for determining a target and a plurality of database records, the method further comprising:

determining for each database record the probability of duplication or non-duplication as a probability that the record is the target of the search criteria; and

displaying in a graphical user interface the database records and a corresponding probability.

8. The computer-implemented method of claim 1, wherein the record pair is a search criteria for determining a target and a plurality of database records, the method further comprising:

determining for each database record the probability of duplication or non-duplication as a confidence score corresponding to the search criteria; and

displaying in a graphical user interface each database records and a corresponding confidence score.

9. A computer-implemented method comprising:

receiving a record pair; and

outputting a probability of duplication between the record pair from an observation of field values of the record pair and noisy characteristics of the record pair.

10. The computer-implemented method of claim 9, wherein the observation of field values is one of an edit distance, a soundex distance, a numerical distance, or a date distance between a pair of fields corresponding to the record pair, respectively.

11. The computer-implemented method of claim 9, further comprising modeling the noisy characteristics of the record pair comprising:

determining a probability of a difference between attribute values corresponding to the fields; and

determining a probability of an error in the field values.

19. The method of claim 11, wherein the record pair is a search criteria for determining a target and a plurality of database records, the method further comprising:

determining for each database record the probability of duplication or non-duplication as a confidence score corresponding to the search criteria; and

displaying in a graphical user interface each database records and a corresponding confidence score.

12. A program storage device readable by machine, tangibly embodying a program of instructions executable by the machine to perform method steps for probabilistic record linkage, the method steps comprising:

providing a record pair comprising a plurality of fields;

providing a plurality of scenarios, each scenario relating to a distribution of patterns among a plurality of attribute statuses;

comparing the record pair to determine a record difference;

determining a probability of a status for each of a plurality of attributes based on the distance metric of the plurality of fields, wherein each field corresponds to a respective attribute, wherein the field is observable and the attribute is hidden;

determining a probability of each scenario based on the probability of the status for each attribute and the Bayesian net representing the probabilistic model on the relationship between scenarios and attributes; and

outputting a probability of duplication or non-duplication of the record pair determined from the probabilities of the plurality of scenarios.

13. The method of claim 12, wherein comparing the record pair comprises comparing record values of the record pair field-wise or across fields.

14. The method of claim 12, wherein determining the probability of a status for each of the plurality of attributes comprises:

providing a predefined error rate of data entering in a field;

determining a distance metric between field values; and

determining a probability of making i errors when entering m characters with the predefined error rate.

15. The method of claim 12, wherein each among a plurality of scenarios is characterized by a probability model on patterns of attribute statuses for example Bayesian net, conditional probabilities of attribute status given scenarios.

16. The method of claim 12, wherein the probability of duplication is compared to a threshold, wherein the threshold corresponds to a significant probability of duplication.

17. The method of claim 12, further comprising:

providing a graphical user interface; and

displaying at least one of a scenario probability, a most probable scenario, a probability of duplication, and/or a probability that an entity is intended by an input search criteria.

18. The method of claim 12, wherein the record pair is a search criteria for determining a target and a plurality of database records, the method further comprising:

determining for each database record the probability of duplication or non-duplication as a probability that the record is the target of the search criteria; and

displaying in a graphical user interface the database records and a corresponding probability.