DATA STREAM QUALITY MANAGEMENT FOR ANALYTIC ENVIRONMENTS

• US 20130297628A1
• Filed: 02/11/2013
• Published: 11/07/2013
• Est. Priority Date: 05/04/2012
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: George, Randy, McKeown, Robert J.
• Granted Patent: US 9,690,812 B2
• Time in Patent Office: Days
• US Class Current: 707/756
• CPC Class Codes: G06F 16/215 Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
• CPC Class Codes: G06F 16/22 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/2365 Ensuring data consistency and integrity
• CPC Class Codes: G06F 16/24568 Data stream processing; Continuous queries
• CPC Class Codes: G06F 40/169 Annotation, e.g. comment data or footnotes

Abstract

According to one aspect of the present disclosure, a method and technique for data quality management is disclosed. The method includes: deploying, into a runtime environment with a data stream analytic module, an ingress quality specification (IQS) module associated with the analytic module; receiving, by the IQS module, the data stream; analyzing, by the IQS module, a subset of data of the data stream to determine if the subset of data meets a quality expectation of the analytic module; annotating the subset of data to indicate a quality status based on whether the subset of data meets the quality expectation of the analytic module; and outputting the data stream to the analytic module.

Claims

1. A method, comprising:

deploying, into a runtime environment with a data stream analytic module, an ingress quality specification (IQS) module associated with the analytic module;

receiving, by the IQS module, the data stream;

analyzing, by the IQS module, a subset of data of the data stream to determine if the subset of data meets a quality expectation of the analytic module;

annotating the subset of data to indicate a quality status based on whether the subset of data meets the quality expectation of the analytic module; and

outputting the data stream to the analytic module.

2. The method of claim 1, further comprising:

receiving, by the analytic module, the data stream;

determining, by the analytic module, the quality status of the subset of data; and

responsive to determining that the quality status indicates that the subset of data does not meet the quality expectation, omitting the subset of data from an analysis of the data stream by the analytic module.

3. The method of claim 1, further comprising receiving a selection of the IQS module from a plurality of IQS modules associated with the analytic module.

4. The method of claim 1, further comprising:

selecting, by the IQS module, a portion of the data stream as the subset of data; and

applying, by the IQS module, a predicate to the subset of data.

5. The method of claim 4, further comprising flagging data of the subset passing the predicate.

6. The method of claim 5, further comprising flagging data of the subset failing the predicate.