Determining and validating provenance data in data stream processing system

• US 8,775,344 B2
• Filed: 05/22/2008
• Issued: 07/08/2014
• Est. Priority Date: 05/22/2008
• Status: Active Grant

• Current Assignee: International Business Machines Corporation, National Research Foundation Of Korea
• Original Assignee: International Business Machines Corporation
• Inventors: Blount, Marion Lee, Davis, John Sidney II, Ebling, Maria Rene, Misra, Archan, Sow, Daby Mousse, Wang, Min
• Primary Examiner(s): Chen, Alan
• Assistant Examiner(s): Bharadwaj, Kalpana
• Application Number:
  US12/125,219
• Publication Number:
  US 20090292818A1
• Time in Patent Office: 2,238 Days
• Field of Search: 706/45
• US Class Current: 706/45
• CPC Class Codes: G06F 15/16 Combinations of two or more digital computers each having at least an arithmetic unit, a program unit and a register, e.g. for a simultaneous processing of several programs coordinating program control therefor G06F9/52; in regulating and control system G05B
• CPC Class Codes: G06F 18/00 Pattern recognition
• CPC Class Codes: G06F 2218/00 Aspects of pattern recognition specially adapted for signal processing
• CPC Class Codes: G16H 50/20 for computer-aided diagnosis, e.g. based on medical expert systems

Abstract

Techniques are disclosed for determining and validating provenance data in such data stream processing systems. For example, a method for processing data associated with a data stream received by a data stream processing system, wherein the system comprises a plurality of processing elements, comprises the following steps. Input data elements and output data elements associated with at least one processing element of the plurality of processing elements are obtained. One or more intervals are computed for the processing element using data representing observations of associations between inputs elements and output elements of the processing element, wherein, for a given one of the intervals, one or more particular input elements contained within the given interval are determined to have contributed to a particular output element. In another method, intervals are specified, and then validated by comparing the specified intervals against intervals computed based on observations.

Claims

1. A method for processing data associated with a data stream received by a data stream processing system, wherein the system comprises a plurality of processing elements, the method comprising the steps of:

obtaining a data stream of input elements and a data stream of output elements associated with at least one processing element of the plurality of processing elements, wherein the data stream of input elements are obtained from at least one streaming data source, and wherein the data stream of output elements are generated by the at least one processing element in response to the data stream of input elements;

computing one or more intervals for the at least one processing element, wherein the one or more intervals are computed using data representing observations of associations between the input elements and the output elements of the at least one processing element, wherein, for a given one of the computed intervals, one or more particular input elements contained within the given computed interval are determined to have contributed to a particular output element; and

using the computed one or more intervals to determine a dependency function that enables a provenance of the particular output element to be determined in terms of the one or more particular input elements.

2. The method of claim 1, wherein the one or more intervals are inferred offline using previously observed and stored sets of input elements and corresponding output elements.

3. The method of claim 1, wherein the one or more intervals are inferred online by using sets of newly observed input elements and corresponding output elements to continually increase the observed data being used, and continually refining the computation of the intervals to utilize the additional observed data.

4. The method of claim 1, wherein the one or more intervals are inferred using a combination of offline and online techniques.

5. The method of claim 1, wherein the observed data comprises actual input and output elements generated by external processes or events, as part of a normal operation of the data stream processing system.

6. The method of claim 1, wherein the observed data comprises input elements that are generated for one or more probe purposes as part of the interval computing step, and their corresponding output elements.

7. The method of claim 1, wherein the one or more intervals are computed using one or more statistical analysis methods over sets of input elements and their corresponding output elements generated by the data stream processing system.

8. The method of claim 7, wherein one of the statistical analysis methods uses entropy-based computations on empirical joint distributions of the input and output elements.

9. The method of claim 1, wherein the one or more intervals are computed using one or more linear transformations over sets of input elements and their corresponding output elements generated by the data stream processing system.

10. The method of claim 1, wherein the interval computing step further comprises using a specified confidence parameter that determines a length of the interval established.

11. The method of claim 10, wherein the confidence parameter is specified as part of a query.

12. The method of claim 1, wherein the one or more computed intervals comprise one or more time intervals computed on the input elements that affect the generation of the corresponding output elements.

13. The method of claim 1, wherein the one or more computed intervals comprise one or more sequence intervals computed on the input elements that affect the generation of the corresponding output elements.

14. The method of claim 1, wherein the step of using observations of associations between input and output elements further comprises using all of the input and output elements that have been observed within a specified past history.

15. The method of claim 1, wherein the step of using observations of associations between input and output elements further comprises using, for each output element, only a subset of the corresponding input elements, wherein the subset is obtained via filtering the total set of input elements using an externally specified filtering criterion.

16. A method for processing data associated with a data stream received by a data stream processing system, wherein the system comprises a plurality of processing elements, the method comprising the steps of:

obtaining input data elements and output data elements associated with at least one processing element of the plurality of processing elements, wherein the input data elements are obtained from at least one streaming data source;

specifying one or more intervals for the processing element wherein, for a given one of the intervals, one or more particular input elements contained within the given interval are believed to have contributed to a particular output element thereby determining a provenance of the particular output element in terms of the one or more particular input elements; and

validating the one or more specified intervals by computing one or more intervals for the processing element using data representing observations of associations between inputs elements and output elements of the processing element, and comparing the one or more specified intervals and the one or more computed intervals.

17. The method of claim 16, further comprising the step of generating an alert notification when a difference between comparing the one or more specified intervals and the one or more computed intervals exceeds a specified threshold.

18. Apparatus for processing data associated with a data stream received by a data stream processing system, wherein the system comprises a plurality of processing elements, the apparatus comprising:

a memory; and

a processor coupled to the memory and configured to;

obtain a data stream of input elements and a data stream of output elements associated with at least one processing element of the plurality of processing elements, wherein the data stream of input elements are obtained from at least one streaming data source, and wherein the data stream of output elements are generated by the at least one processing element in response to the data stream of input elements;

compute one or more intervals for the at least one processing element, wherein the one or more intervals are computed using data representing observations of associations between the input elements and the output elements of the at least one processing element, wherein, for a given one of the computed intervals, one or more particular input elements contained within the given computed interval are determined to have contributed to a particular output element; and

use the computed one or more intervals to determine a dependency function that enables a provenance of the particular output element to be determined in terms of the one or more particular input elements.

19. The method of claim 18, wherein the one or more computed intervals comprise at least one of:

one or more time intervals computed on the input elements that affect the generation of the corresponding output elements; and

one or more sequence intervals computed on the input elements that affect the generation of the corresponding output elements.

20. An article of manufacture for processing data associated with a data stream received by a data stream processing system, wherein the system comprises a plurality of processing elements, the article comprising a computer readable storage medium having one or more programs embodied therewith wherein the one or more programs, when executed by a computer, perform steps of:

obtaining a data stream of input elements and a data stream of output elements associated with at least one processing element of the plurality of processing elements, wherein the data stream of input elements are obtained from at least one streaming data source, and wherein the data stream of output elements are generated by the at least one processing element in response to the data stream of input elements;

computing one or more intervals for the at least one processing element, wherein the one or more intervals are computed using data representing observations of associations between the input elements and the output elements of the at least one processing element, wherein, for a given one of the computed intervals, one or more particular input elements contained within the given computed interval are determined to have contributed to a particular output element; and

using the computed one or more intervals to determine a dependency function that enables a provenance of the particular output element to be determined in terms of the one or more particular input elements.