Method and Apparatus for Accelerated Data Quality Checking

• US 20130151458A1
• Filed: 02/05/2013
• Published: 06/13/2013
• Est. Priority Date: 05/15/2008
• Status: Active Grant

• Current Assignee: Ip Reservoir, LLC
• Original Assignee: Exegy Incorporated
• Inventors: Indeck, Ronald S., Indeck, David Mark, Singla, Naveen, White, Jason R.
• Granted Patent: US 9,547,824 B2
• Time in Patent Office: Days
• US Class Current: 706/47
• CPC Class Codes: G06F 16/24544 Join order optimisation
• CPC Class Codes: G06F 16/24568 Data stream processing; Continuous queries
• CPC Class Codes: G06F 9/542 Event management; Broadcasting; Multicasting; Notifications
• CPC Class Codes: G06N 5/025 Extracting rules from data
• CPC Class Codes: H03M 13/00 Coding, decoding or code conversion, for error detection or error correction; Coding theory basic assumptions; Coding bounds; Error probability evaluation methods; Channel models; Simulation or testing of codes error detection or error correction for analogue/digital, digital/analogue or code conversion H03M1/00 – H03M11/00; specially adapted for digital computers G06F11/08; for information storage based on relative movement between record carrier and transducer G11B, e.g. G11B20/18; for static stores G11C

Abstract

Disclosed herein is a method and apparatus for hardware-accelerating various data quality checking operations. Incoming data streams can be processed with respect to a plurality of data quality check operations using offload engines (e.g., reconfigurable logic such as field programmable gate arrays (FPGAs)). Accelerated data quality checking can be highly advantageous for use in connection with Extract, Transfer, and Load (ETL) systems.

Claims

1. A method of integrating a data stream within an enterprise computing system, the data stream comprising a plurality of records, each record having at least one data field, each data field having data therein, the method comprising:

processing the data stream with a plurality of hardware-accelerated data quality check operations, the data quality check operations corresponding to a plurality of rule conditions for the data fields and being configured to determine whether the data within the data fields of the data stream satisfy any of the rule conditions; and

generating a plurality of rule condition check results for the records based on the processing step.

2. The method of claim 1 further comprising:

receiving the data stream within the enterprise computing system, and

performing the processing step and the generating step before storing the records in a database within the enterprise computing system.

3. The method of claim 2 wherein at least one of the hardware-accelerated rule condition check operations comprises a hardware-accelerated range check operation.

4. The method of claim 2 wherein at least one of the hardware-accelerated rule condition check operations comprises a hardware-accelerated matching operation.

5. The method of claim 2 wherein the processing step further comprises performing a hardware-accelerated derived value computation operation to compute a derived value for at least one record, and wherein the hardware-accelerated rule condition check operation is based on the computed derived value.

6. The method of claim 2 wherein the processing step further comprises performing a hardware-accelerated aggregate value computation operation to compute an aggregate value for at least one record, and wherein the hardware-accelerated rule condition check operation is based on the computed aggregate value.

7. The method of claim 2 further comprising performing a hardware-accelerated filtering operation on the data stream upstream from the hardware-accelerated rule condition check operation.

8. The method of claim 1 wherein the data stream comprises a plurality of XML data records.

9. The method of claim 1 wherein the data stream comprises a plurality of StreamSQL data records.

10. The method of claim 1 wherein the processing step and the generating step are performed by a reconfigurable logic device.

11. The method of claim 1 wherein the processing step and the generating step are performed by a graphics processor unit (GPU).

12. The method of claim 1 wherein the computer system comprises a reconfigurable logic device, the reconfigurable logic device having a pipeline deployed thereon, the pipeline comprising a range check module, wherein the processing step comprises the range check module determining whether the data in a data field of interest in a record falls within a defined range of data values, and wherein the generating step comprises the range check module generating a rule condition check result indicative of whether the data in the data field of interest falls within the defined range in response to the determining step.

13. The method of claim 12 wherein the pipeline further comprises a filter module upstream from the range check module, and wherein the processing step comprises the filter module (1) identifying a plurality of data fields for records in the data stream that have a range constraint, and (2) providing the identified data fields to the range check module for processing thereby as data fields of interest.

14. The method of claim 13 wherein the processing step and the generating step are performed on records within the data stream before those records are stored in a database within the enterprise computing system.

15. The method of claim 12 wherein the range check module comprises a first comparator and a second comparator, and wherein the determining step comprises:

the range check module accessing a data table to retrieve a lower limit and an upper limit for the defined range;

the first comparator comparing the data in the data field of interest with the retrieved lower limit to assess whether the data in the data field of interest is less than the lower limit; and

the second comparator comparing the data in the data field of interest with the retrieved upper limit to assess whether the data in the data field of interest is greater than the upper limit; and

wherein the first comparator and second comparator perform their respective comparing steps in parallel with each other; and

wherein the generating step comprises the range check module, in response to a determination that the data in the data field of interest is less than the lower limit or greater than the upper limit, identifying the data in the data field of interest as being in an out of range condition.

16. The method of claim 1 wherein the data in the data fields of the records are expressed by a plurality of characters, wherein the computer system comprises a reconfigurable logic device, the reconfigurable logic device having a pipeline deployed thereon, the pipeline comprising a character check module, wherein the processing step comprises the character check module determining whether any of the characters in a data field of interest in a record are not members of a defined character set, and wherein the generating step comprises the character check module generating a rule condition check result indicative of whether the characters in the data field of interest are members of the defined character set in response to the determining step.

17. The method of claim 16 wherein the pipeline further comprises a filter module upstream from the character check module, and wherein the processing step comprises the filter module (1) identifying a plurality of data fields for records in the data stream that have a character set constraint, and (2) providing the identified data fields to the character check module for processing thereby as data fields of interest.

18. The method of claim 17 wherein the processing step and the generating step are performed on records within the data stream before those records are stored in a database within the enterprise computing system.

19. The method of claim 16 wherein the pipeline further comprises a character parsing module upstream from the character check module, and wherein the processing step further comprises:

the character parsing module parsing the data field of interest into its constituent characters.

20. The method of claim 1 wherein the computer system comprises a reconfigurable logic device, the reconfigurable logic device having a pipeline deployed thereon, the pipeline comprising an exact matching module, wherein the processing step comprises the exact matching module determining whether the data in a data field of interest in a record is a member of a defined value set, and wherein the generating step comprises the exact matching module generating a rule condition check result indicative of whether the data in the data field of interest is a member of the defined value set in response to the determining step.

21. The method of claim 20 wherein the pipeline further comprises a filter module upstream from the exact matching module, and wherein the processing step comprises the filter module (1) identifying a plurality of data fields for records in the data stream that have a value set constraint, and (2) providing the identified data fields to the exact matching module for processing thereby as data fields of interest.

22. The method of claim 21 wherein the processing step and the generating step are performed on records within the data stream before those records are stored in a database within the enterprise computing system.

23. The method of claim 20 further comprising:

keying the exact matching module with values that are members of the value set; and

wherein the determining step comprises the exact matching module comparing the data in the data field of interest with the keyed values; and

wherein the generating step comprises, in response to the comparing step resulting in a determination that the data in the data field of interest does not match any of the keyed values, identifying the data in the data field of interest as being in an out of value condition.

24. The method of claim 1 wherein the computer system comprises a reconfigurable logic device, the reconfigurable logic device having a pipeline deployed thereon, the pipeline comprising a plurality of parallel processing paths, and wherein the processing step comprises the parallel processing paths performing different data quality check operations on the data stream in parallel.

25. The method of claim 24 wherein the data in the data fields of the records are expressed by a plurality of characters, wherein the parallel processing paths comprise a first processing path and a second processing path, and wherein the processing step further comprises:

the first processing path (1) filtering the data stream to identify data fields for records in the data stream that have a range constraint, and (2) performing range check operations on the data in the identified data fields having the range constraint to generate data indicative of whether the identified data fields having the range constraint comply with the range constraint; and

the second processing path (1) filtering the data stream to identify data fields for records in the data stream that have a character set constraint, and (2) performing character check operations on the characters in the identified data fields having the character set constraint to generate data indicative of whether the identified data fields having the character set constraint comply with the character set constraint; and

wherein the first processing path and the second processing path perform their respective operations in parallel at hardware processing speeds.

26. The method of claim 25 wherein the parallel processing paths further comprise a third processing path, and wherein the processing step further comprises:

the third processing path (1) filtering the data stream to identify data fields for records in the data stream that have a value set constraint, and (2) performing value check operations on the data in the identified data fields having the value set constraint to generate data indicative of whether the identified data fields having the value set constraint comply with the value set constraint; and

wherein the first processing path, the second processing path, and the third processing path perform their respective operations in parallel at hardware processing speeds.

27. The method of claim 25 wherein the parallel processing paths perform their respective operations on records within the data stream before those records are stored in a database within the enterprise computing system.

28. The method of claim 27 wherein the reconfigurable logic device comprises a field programmable gate array (FPGA), the pipeline being deployed on the FPGA.

29. The method of claim 1 wherein the computer system comprises a graphics processor unit (GPU), the GPU having a pipeline deployed thereon, the pipeline comprising a plurality of parallel processing paths, and wherein the processing step comprises the parallel processing paths performing different data quality check operations on the data stream in parallel.

30. An apparatus for integrating a data stream within an enterprise computing system, the data stream comprising a plurality of records, each record having at least one data field, each data field having data therein, the apparatus comprising:

a coprocessor configured to (1) process the data stream with a plurality of hardware-accelerated data quality check operations, the data quality check operations corresponding to a plurality of rule conditions for the data fields and being configured to determine whether the data within the data fields of the data stream satisfy any of the rule conditions, and (2) generate a plurality of rule condition check results for the records based on the processing operation.

31. The apparatus of claim 30 wherein the coprocessor comprises a reconfigurable logic device, the reconfigurable logic device having a pipeline deployed thereon, the pipeline comprising a range check module, wherein the range check module is configured to (1) determine whether the data in a data field of interest in a record falls within a defined range of data values, and (2) generate a rule condition check result indicative of whether the data in the data field of interest falls within the defined range in response to the determination.

32. The apparatus of claim 31 wherein the pipeline further comprises a filter module upstream from the range check module, the filter module configured to (1) identify a plurality of data fields for records in the data stream that have a range constraint, and (2) provide the identified data fields to the range check module for processing thereby as data fields of interest.

33. The apparatus of claim 32 wherein the reconfigurable logic device is configured to process records within the data stream through the pipeline before those records are stored in a database within the enterprise computing system.

34. The apparatus of claim 31 wherein the range check module comprises a first comparator and a second comparator, wherein the range check module is further configured to access a data table to retrieve a lower limit and an upper limit for the defined range, wherein the first comparator is configured to compare the data in the data field of interest with the retrieved lower limit to assess whether the data in the data field of interest is less than the lower limit, wherein the second comparator is configured to compare the data in the data field of interest with the retrieved upper limit to assess whether the data in the data field of interest is greater than the upper limit, wherein the first comparator and second comparator are configured to perform their respective comparisons in parallel with each other, and wherein the range check module is further configured to, in response to a determination that the data in the data field of interest is less than the lower limit or greater than the upper limit, identify the data in the data field of interest as being in an out of range condition.

35. The apparatus of claim 30 wherein the data in the data fields of the records are expressed by a plurality of characters, wherein the coprocessor comprises a reconfigurable logic device, the reconfigurable logic device having a pipeline deployed thereon, the pipeline comprising a character check module, wherein the character check module is configured to (1) determine whether any of the characters in a data field of interest in a record are not members of a defined character set, and (2) generate a rule condition check result indicative of whether the characters in the data field of interest are members of the defined character set in response to the determination.

36. The apparatus of claim 35 wherein the pipeline further comprises a filter module upstream from the character check module, and wherein the filtering module is configured to (1) identify a plurality of data fields for records in the data stream that have a character set constraint, and (2) provide the identified data fields to the character check module for processing thereby as data fields of interest.

37. The apparatus of claim 36 wherein the reconfigurable logic device is configured to process records within the data stream through the pipeline before those records are stored in a database within the enterprise computing system.

38. The apparatus of claim 35 wherein the pipeline further comprises a character parsing module upstream from the character check module, and wherein the character parsing module is configured to parse the data field of interest into its constituent characters.

39. The apparatus of claim 30 wherein the coprocessor comprises a reconfigurable logic device, the reconfigurable logic device having a pipeline deployed thereon, the pipeline comprising an exact matching module, wherein exact matching module is configured to (1) determine whether the data in a data field of interest in a record is a member of a defined value set, and (2) generate a rule condition check result indicative of whether the data in the data field of interest is a member of the defined value set in response to the determination.

40. The apparatus of claim 39 wherein the pipeline further comprises a filter module upstream from the exact matching module, and wherein the filter module is configured to (1) identify a plurality of data fields for records in the data stream that have a value set constraint, and (2) provide the identified data fields to the exact matching module for processing thereby as data fields of interest.

41. The apparatus of claim 40 wherein the reconfigurable logic device is configured to process records within the data stream through the pipeline before those records are stored in a database within the enterprise computing system.

42. The apparatus of claim 39 wherein the exact matching module is keyed with values that are members of the value set, wherein the exact matching module is further configured to (1) compare the data in the data field of interest with the keyed values, and (2) in response to the comparison resulting in a determination that the data in the data field of interest does not match any of the keyed values, identify the data in the data field of interest as being in an out of value condition.

43. The apparatus of claim 30 wherein the coprocessor comprises a reconfigurable logic device, the reconfigurable logic device having a pipeline deployed thereon, the pipeline comprising a plurality of parallel processing paths, and wherein the parallel processing paths are configured to perform different data quality check operations on the data stream in parallel.

44. The apparatus of claim 43 wherein the data in the data fields of the records are expressed by a plurality of characters, wherein the parallel processing paths comprise a first processing path and a second processing path;

wherein the first processing path comprises a filter module and a downstream range check module, the filter module in the first processing path configured to filter the data stream to identify data fields for records in the data stream that have a range constraint, and the range check module configured to perform range check operations on the data in the identified data fields having the range constraint to generate data indicative of whether the identified data fields having the range constraint comply with the range constraint;

wherein the second processing path comprises a filter module and a downstream character check module, the filter module in the second processing path configured to filter the data stream to identify data fields for records in the data stream that have a character set constraint, and the character check module configured to perform character check operations on the characters in the identified data fields having the character set constraint to generate data indicative of whether the identified data fields having the character set constraint comply with the character set constraint; and

wherein the first processing path and the second processing path are configured to perform their respective operations in parallel at hardware processing speeds.

45. The apparatus of claim 44 wherein the parallel processing paths further comprise a third processing path, wherein the third processing path comprises a filter module and a downstream exact matching module, the filter module in the third processing path configured to filter the data stream to identify data fields for records in the data stream that have a value set constraint, the exact matching module configured to perform value check operations on the data in the identified data fields having the value set constraint to generate data indicative of whether the identified data fields having the value set constraint comply with the value set constraint;

and

wherein the first processing path, the second processing path, and the third processing path are configured to perform their respective operations in parallel at hardware processing speeds.

46. The apparatus of claim 44 wherein the parallel processing paths are configured to perform their respective operations on records within the data stream before those records are stored in a database within the enterprise computing system.

47. The apparatus of claim 46 wherein the reconfigurable logic device comprises a field programmable gate array (FPGA), the pipeline being deployed on the FPGA.

48. The apparatus of claim 30 wherein the coprocessor comprises a graphics processor unit (GPU), the GPU having a pipeline deployed thereon, the pipeline comprising a plurality of parallel processing paths, and wherein the parallel processing paths are configured to perform different data quality check operations on the data stream in parallel.

49. A method of preprocessing a data stream for an extract, transfer, and load (ETL) operation with respect to a data store, the method comprising:

receiving streaming data at a reconfigurable logic device, the reconfigurable logic device having a pipeline deployed thereon, the pipeline comprising a plurality of parallel processing paths;

the parallel processing paths of the pipeline performing different data quality check operations on the streaming data in parallel to generate data indicative of whether the streaming data satisfies predetermined criteria; and

downstream from the pipeline, loading the streaming data into a data store.

50. The method of claim 49 wherein the streaming data comprises a plurality of records, each record comprising at least one data field, and wherein the performing step comprises:

performing, in the parallel processing paths of the pipeline, at least two members of the group consisting of (1) a range check operation, (2) a character check operation, (3) a value check operation, and (4) a bit length check operation on select data fields of the streaming data at hardware processing speeds.

51. The method of claim 50 wherein the data store comprises an enterprise database.

52. An apparatus for preprocessing a data stream for an extract, transfer, and load (ETL) operation with respect to a data store, the apparatus comprising:

a processor;

a reconfigurable logic device; and

a data store;

wherein the reconfigurable logic device is configured to receive streaming data, the reconfigurable logic device having a pipeline deployed thereon, the pipeline comprising a plurality of parallel processing paths;

wherein the parallel processing paths of the pipeline are configured to perform different data quality check operations on the streaming data in parallel to generate data indicative of whether the streaming data satisfies predetermined criteria; and

wherein the processor is configured to, downstream from the pipeline, load the streaming data into the data store.

53. The apparatus of claim 52 wherein the streaming data comprises a plurality of records, each record comprising at least one data field, and wherein the parallel processing paths of the pipeline are configured to perform at least two members of the group consisting of (1) a range check operation, (2) a character check operation, (3) a value check operation, and (4) a bit length check operation on select data fields of the streaming data at hardware processing speeds.

54. The apparatus of claim 53 wherein the data store comprises an enterprise database.