Data stream quality management for analytic environments

• US 9,690,812 B2
• Filed: 02/11/2013
• Issued: 06/27/2017
• Est. Priority Date: 05/04/2012
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: George, Randy, McKeown, Robert J.
• Primary Examiner(s): Hutton, Nan
• Application Number:
  US13/764,696
• Publication Number:
  US 20130297628A1
• Time in Patent Office: 1,597 Days
• CPC Class Codes: G06F 16/215 Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
• CPC Class Codes: G06F 16/22 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/2365 Ensuring data consistency and integrity
• CPC Class Codes: G06F 16/24568 Data stream processing; Continuous queries
• CPC Class Codes: G06F 40/169 Annotation, e.g. comment data or footnotes

Abstract

According to one aspect of the present disclosure, a method and technique for data quality management is disclosed. The method includes: deploying, into a runtime environment with a data stream analytic module, an ingress quality specification (IQS) module associated with the analytic module; receiving, by the IQS module, the data stream; analyzing, by the IQS module, a subset of data of the data stream to determine if the subset of data meets a quality expectation of the analytic module; annotating the subset of data to indicate a quality status based on whether the subset of data meets the quality expectation of the analytic module; and outputting the data stream to the analytic module.

Claims

1. A method, comprising:

deploying, into a runtime environment upstream from an analytic module, an ingress quality specification (IQS) module;

receiving, by the IQS module, a data stream;

analyzing, by the IQS module, a subset of data of the data stream to determine if the subset of data meets a quality expectation of the analytic module;

annotating the subset of data to indicate a quality status based on whether the subset of data meets the quality expectation of the analytic module;

outputting the data stream with the annotated subset of data to the analytic module;

receiving, by the analytic module, the data stream, the analytic module analyzing the data stream to assess an operating characteristic of an upstream device;

determining, by the analytic module, the quality status of the subset of data; and

responsive to determining that the quality status indicates that the subset of data does not meet the quality expectation, omitting the subset of data from an analysis of the data stream by the analytic module.

2. The method of claim 1, further comprising receiving a selection of the IQS module from a plurality of IQS modules associated with the analytic module.

3. The method of claim 1, further comprising:

selecting, by the IQS module, a portion of the data stream as the subset of data; and

applying, by the IQS module, a predicate to the subset of data.

4. The method of claim 3, further comprising flagging data of the subset passing the predicate.

5. The method of claim 4, further comprising flagging data of the subset failing the predicate.

6. The method of claim 1, further comprising determining, by the IQS module, whether the selected subset of data of the data stream includes a minimum quantity of data samples based on the quality expectation of the analytic module.

7. A method, comprising:

deploying, into a runtime environment an analytic module, the analytic module configured to analyze a data stream associated with an upstream object and output an analysis of the object;

deploying, into the runtime environment upstream from the analytic module, an ingress quality specification (IQS) module;

receiving, by the IQS module, the data stream;

analyzing, by the IQS module, a subset of data of the data stream to determine if the subset of data meets a quality expectation of the analytic module;

annotating the subset of data via an additional field of the data to indicate a quality status based on whether the subset of data meets the quality expectation of the analytic module;

outputting the data stream with the annotated subset of data to the analytic module;

receiving, by the analytic module, the data stream;

determining, by the analytic module, the quality status of the subset of data; and

responsive to determining that the quality status indicates that the subset of data does not meet the quality expectation, omitting the subset of data from an analysis of the data stream by the analytic module.

8. The method of claim 7, further comprising receiving a selection of the IQS module from a plurality of IQS modules associated with the analytic module.

9. The method of claim 7, further comprising:

selecting, by the IQS module, a portion of the data stream as the subset of data; and

applying, by the IQS module, a predicate to the subset of data.

10. The method of claim 9, further comprising flagging data of the subset passing the predicate.

11. The method of claim 10, further comprising flagging data of the subset failing the predicate.

12. A method, comprising:

deploying, into a runtime environment with a data stream analytic module, an ingress quality specification (IQS) module associated with the analytic module;

receiving, by the IQS module, the data stream;

analyzing, by the IQS module, a subset of data of the data stream to determine if the subset of data meets a quality expectation of the analytic module;

annotating the subset of data to indicate a quality status based on whether the subset of data meets the quality expectation of the analytic module;

outputting the data stream with the annotated subset of data to the analytic module;

receiving, by the analytic module, the data stream from the IQS module;

identifying, by the analytic module, data not meeting the quality expectation based on the annotations; and

omitting, by the analytic module, from its analysis of the subset data the data not meeting the quality expectation.

13. The method of claim 12, further comprising receiving a selection of the IQS module from a plurality of IQS modules associated with the analytic module.

14. The method of claim 12, further comprising:

selecting, by the IQS module, a portion of the data stream as the subset of data; and

applying, by the IQS module, a predicate to the subset of data.

15. The method of claim 14, further comprising flagging data of the subset passing the predicate.

16. The method of claim 15, further comprising flagging data of the subset failing the predicate.