Systems and methods for record linkage and paraphrase generation using surrogate learning

• US 8,108,326 B2
• Filed: 02/06/2009
• Issued: 01/31/2012
• Est. Priority Date: 02/06/2008
• Status: Active Grant

• Current Assignee: Thomson Reuters Enterprise Centre GmbH (The Woodbridge Co. Ltd.)
• Original Assignee: Thomson Reuters Global Resources
• Inventors: Veeramachaneni, Sriharsha
• Primary Examiner(s): Gaffin, Jeffrey A
• Assistant Examiner(s): OLUDE AFOLABI, OLATOYOSI
• Application Number:
  US12/367,371
• Publication Number:
  US 20090228410A1
• Time in Patent Office: 1,089 Days
• Field of Search: 706/12, 382/224
• US Class Current: 706/12
• CPC Class Codes: G06F 16/10 File systems; File servers
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks

Abstract

A method of using unlabeled data to train a classifier is disclosed. In one embodiment related to record linkage, the method entails retrieving a set of candidate data records from a master database based on a least one update record. Next, a surrogate learning technique is used to identify one of the candidate data records as a match for the one update record. Lastly, the exemplary method links or merges the update record and the identified one of the candidate data records.

Claims

1. A method of using a processor and a memory for classifying data associated with a feature space X to a set of classes y={0,1}, wherein features defining the feature space X are partitioned into X=X1×

X2, a random feature vector xε

X is denoted correspondingly as x=(x1, x2), and feature x1 is a binary random variable, the method comprising;

estimating P(x1|x2) from a set of unlabeled data;

estimating P(x1=0|x2) from a set of labeled data;

determining whether to classify a portion of the data to y=0 or y=1 based on the estimated P(x1=0|x2); and

logically associating the portion of the data in the memory with the class y=0 or the class y=1 based on the determination.

2. The method of claim 1, wherein the unlabeled data is a set of identity records, and the binary random variable indicates whether a field of one of the identity records matches a corresponding field in another identity record.

3. The method of claim 1, wherein the feature x1 is a total recall feature for the class y=1.

4. The method of claim 1, wherein P(x1,x2|y=0)=P(x1|y=0) P(x2|y=0).

5. The method of claim 1, wherein the data is a segment of text from a new article.

6. A system having a processor and a memory for classifying data associated with a feature space X to a set of classes y={0,1}, wherein features defining the feature space X are partitioned into X=X1×

X2, a random feature vector xε

X is denoted correspondingly as x=(x1, x2), and feature x1 is a binary random variable, the system further comprising;

means for estimating P(x1|x2) from a set of unlabeled data;

means for estimating P(x1=0|x2) from a set of labeled data;

means for determining whether to classify a portion of the data to y=0 or y=1 based on the estimated P(x1=0\x2); and

means, responsive to the determination, for logically associating the portion of the data in the memory with the class y=0 or the class y=1.

7. The system of claim 6, wherein the unlabeled data is a set of identity records, and the binary random variable indicates whether a field of one of the identity records matches a corresponding field in another identity record.

8. The system of claim 6, wherein the feature x1 is a total recall feature for the class y=1.

9. The system of claim 6, wherein P(x1,x2|y=0)=P(x1|y=0) P(x2|y=0).

10. A method of using a processor and a memory for linking or merging update records with a master database of data records, the method comprising:

retrieving a set of candidate data records from the master database based on a least one update record;

using surrogate learning to identify one of the candidate data records as a match for the one update record; and

linking or merging the update record and the identified one of the candidate data records.