Boolean rule-based system for clustering similar records

• US 20040107205A1
• Filed: 12/03/2002
• Published: 06/03/2004
• Est. Priority Date: 12/03/2002
• Status: Abandoned Application

• Current Assignee: Lockheed Martin Corporation (Martin Marietta Corporation)
• Original Assignee: Lockheed Martin Corporation (Martin Marietta Corporation)
• Inventors: Szczerba, Robert J., Burdick, Douglas R., Rostedt, Steven
• Application Number:
  US10/308,740
• Publication Number:
  US 20040107205A1
• Time in Patent Office: Days
• US Class Current: 707/102
• CPC Class Codes: G06F 16/215 Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors

Abstract

A system identifies similar records. The system includes a collection of records, a set of Boolean rules, and a cell list structure. Each record in the collection has a list of fields and data contained in each field. The set of Boolean rules operate upon the data in each field. The cell list structure is generated from the collection of records. The cell list structure has a list of cells for each field and a list of pointers to each cell of the list of cells for each record. The set of Boolean rules identifies the similar records from the cell list structure.

Claims

1. A system for identifying similar records, said system comprising:

a collection of records, each said record in said collection having a list of fields and data contained in each said field;

a set of Boolean rules for operating upon the data in each said field;

a cell list structure generated from said collection of records,

said cell list structure having a list of cells for each field and a list of pointers to each said cell of said list of cells for each said record,

said set of Boolean rules identifying the similar records from said cell list structure.

2. The system as set forth in claim 1 wherein said collection of records is formed by parsing data for each said record into fields.

3. The system as set forth in claim 1 wherein said set of Boolean rules operate upon the data during a clustering step of a data cleansing operation.

4. A method for cleansing electronic data, said method comprising the steps of:

inputting a collection of records, each record in the collection representing an entity having a list of fields and data contained in each of the fields;

selecting a plurality of Boolean clustering rules for operating upon the data in each field in each record;

generating a list of clusters by applying the plurality of Boolean clustering rules to the collection of records, the list of clusters comprising a list of candidate duplicate records determined by the plurality of Boolean clustering rules; and

outputting the list of clusters.

5. The method as set forth in claim 4 further includes the step of parsing the data for each said record into fields.

6. The method as set forth in claim 4 further includes the step of correcting errors in the data by reference to a recognized source of correct data.

7. The method as set forth in claim 4 further includes the step of merging the list of candidate duplicate records from said outputting step together to eliminate records representing the same entity.

8. A system for identifying similar records, said system comprising:

a collection of records, each said record in said collection having a list of fields and data contained in each said field;

a set of fuzzy logic rules for operating upon the data in each said field;

a cell list structure generated from said collection of records,

said cell list structure having a list of cells for each field and a list of pointers to each said cell of said list of cells for each said record,

said set of fuzzy logic rules identifying the similar records from said cell list structure.

9. The system as set forth in claim 8 said set of fuzzy logic rules is included in a hierarchy of rules.

10. The system as set forth in claim 9 further including a set of conditions for applying to said hierarchy of rules.