Data tracking arrangement for improving the quality of data stored in a database

• US 5,396,612 A
• Filed: 05/02/1991
• Issued: 03/07/1995
• Est. Priority Date: 05/02/1991
• Status: Expired due to Term

• Current Assignee: Teradata US, Inc. (Teradata Corporation)
• Original Assignee: AT&T Corporation (AT&T, Inc.)
• Inventors: Pautke, Robert W., Redman, Thomas C., Huh, Young U.
• Primary Examiner(s): Beausoliel, Jr., Robert W.
• Assistant Examiner(s): Tu, T.
• Application Number:
  US07/694,573
• Time in Patent Office: 1,405 Days
• Field of Search: 395/575, 364/228, 364/228.3, 364/228.5, 364/228.7, 364/931, 364/931.01, 364/931.4, 364/931.43, 371/5.1
• US Class Current: 714/49
• CPC Class Codes: G06F 16/215 Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
• CPC Class Codes: G06F 2201/80 Database-specific techniques

Abstract

A facility is provided for improving the quality of data that is stored in a data base. Such improved quality is achieved by tracking the operation of various processes controlling respective data processors, which assemble/modify a data record that is ultimately stored in the data base such that random ones of the data records are marked prior to supplying them to a first one of the processors and then obtaining from each of the processors a copy of a marked data record that it processes. Copies of respective data records are then analyzed to determine which of the processes may be corrected to improve the quality of the data that is stored in the data base.

Claims

1. A method of processing data records, each of said data records being assembled by a plurality of data processors each for inserting data received from at least one source of data into particular fields of said data records, said method comprising the steps of

labeling selected ones of said data records as said records are received from said source and supplying said data records in sequence to said data processors, said data processors processing each of said data records in sequence beginning with a first one of said processors,

receiving from each of said data processors copies of said labeled data records as said labeled data records are processed by said data processors and forming said copies into respective series of copies associated with respective ones of said labeled records, and

for each of said series of copies identifying as errors differences in the data inserted in corresponding fields of adjacent ones of said copies, and

is determining, as a function of said errors, an error rate between the processors that processed said adjacent ones of said copies.

2. The method set forth in claim 1 wherein

said errors include normalization, translational or spurious-operational errors, and wherein said error rate is based on spurious-operational errors.

3. A method of processing data records, each of said data records being assembled by a plurality of data processors each for inserting data received from at least one source of data into particular fields of said data records, said method comprising the steps of

labeling selected ones of said data records as said records are received from said source and supplying said data records, in sequence to said data processors, said data processors processing each of said data records in sequence beginning with a first one said processors,

receiving from each of said data processors copies of said labeled data records as said labeled data records are processed by said data processors and forming said copies into respective series of copies associated with respective ones of said labeled records, and for each of said series of copies identifying as errors differences in the data inserted in corresponding fields of adjacent ones of said copies, and

determining, as a function of said errors, an error rate between the processors that processed said adjacent ones of said copies, and wherein said method further comprises the step of

appending to each of said labeled data records a first field for entering at least a time of day that an associated labeled record is received by an individual one of said data processors and a second field for entering at least a time of day that said one data processor outputs to a next one of said data processors said associated labeled record.

4. The method set forth in claim 3 wherein said time of day entered in said first field includes at least an associated date and said time of day entered in said second field includes an associated date.

5. The method set forth in claim 4 wherein said step of appending includes the step of determining a processing time consumed by said one data processor to process said associated labeled data record as a function of the time entered in said first field and the time entered in said second field.

6. Apparatus for processing data records, each of said data records being assembled by a plurality of data processors each for inserting data into particular fields of said data records, said apparatus comprising:

preprocessor means for selecting on a random basis individual ones of said data records and for labeling each of said selected records and supplying said data records in turn to a first one of said data processors so that said records are processed by all of said processors in a predetermined order beginning with said first processor, and

means for receiving from each of said data processors copies of said selected data records as said selected data records are processed by each of said data processors and

for identifying any difference in the data inserted in corresponding fields of copies of the same labeled record received from respective ones of said processors, marking such difference as an error and associating said error with the data processors that caused that difference to occur.

7. The apparatus set forth in claim 6 wherein said means for identifying includes means for determining an error rate between the data processors that caused such difference.

8. The apparatus set forth in claim 6 wherein said error includes normalization, translational or spurious-operational errors, and wherein said error rate is based on spurious-operational errors.

9. Apparatus for processing data records, each of said data records being assembled by a plurality of dater processors each for inserting data into particular fields of said data records, said apparatus comprising:

preprocessor means for selecting on a random basis individual ones of said data records and for labeling each of said selected records and supplying said data records in turn to a first one of said data processors so that said records are processed by all of said processors in a predetermined order beginning with said first processor, and

means for receiving from each of said data processors copies of said selected data records as said selected data records are processed by each of said data processors and

for identifying any difference in the data inserted in corresponding fields of copies of the same labeled record received from respective ones of said processors, marking such difference as an error and associating said error with the data processors that caused that difference to occur, and wherein said apparatus further comprises means for appending to each of said selected data records a first field for entering at least a time of day that said selected records are received by one of said data processors and a second field for entering at least a time of day that said one data processor outputs each of said data records to a next one of said data processors.

10. The apparatus set forth in claim 9 wherein said time of day entered in said first field includes an associated date and said time of day entered in said second field includes an associated date.

11. The apparatus set forth in claim 10 wherein said means for identifying includes means for calculating a processing time consumed by said one data processor to process each of said selected data records as a function of the time of day entered in said first field and the time of day entered in said second field.