PARTIAL FAULT TOLERANT STREAM PROCESSING APPLICATIONS

• US 20110239048A1
• Filed: 03/29/2010
• Published: 09/29/2011
• Est. Priority Date: 03/29/2010
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Jacques da Silva, Gabriela, ANDRADE, HENRIQUE, Wu, Kun-Lung, Gedik, Bugra
• Granted Patent: US 8,458,650 B2
• Time in Patent Office: Days
• US Class Current: 714/35
• CPC Class Codes: G06F 11/28 by checking the correct order of processing G06F11/08 - G06F11/26 take precedence; monitoring patterns of pulse trains H03K5/19
• CPC Class Codes: G06F 11/3612 by runtime analysis performance monitoring G06F11/3466
• CPC Class Codes: G06F 11/3616 using software metrics
• CPC Class Codes: G06F 16/2365 Ensuring data consistency and integrity

Abstract

In one embodiment, the invention comprises partial fault tolerant stream processing applications. One embodiment of a method for implementing partial fault tolerance in a stream processing application comprising a plurality of stream operators includes: defining a quality score function that expresses how well the application is performing quantitatively, injecting a fault into at least one of the plurality of operators, assessing an impact of the fault on the quality score function, and selecting at least one partial fault-tolerant technique for implementation in the application based on the quantitative metric-driven assessment.

Claims

1. A method for implementing partial fault tolerance in a stream processing application comprising a plurality of stream operators, the method comprising:

defining a quality score function that expresses how well the application is performing quantitatively;

injecting a fault into at least one of the plurality of operators;

assessing an impact of the fault on the quality score function; and

selecting at least one partial fault-tolerant technique for implementation in the application based on the assessing.

2. The method of claim 1, wherein the injecting comprises:

modifying source code for the application to produce at least one instance of modified source code, where the at least one instance of modified source code incorporates a fault injection parameter for the at least one partial fault tolerance technique; and

compiling the at least one instance of modified source code into executable code.

3. The method of claim 1, wherein the assessing comprises:

calculating the quality score function using an output of the application in a fault-free condition to generate a target quality score function;

calculating the quality score function using an output of the application including the fault to generate a faulty quality score function; and

determining whether a difference between the faulty quality score function and the target quality score function is within a tolerance threshold.

4. The method of claim 3, wherein the faulty quality score function is calculated over local sections of the output of the application including the fault.

5. The method of claim 1, wherein the quality score function accounts for at least one a metric specific to the at least one of the plurality of operators, and wherein the at least one metric characterizes the application in terms of effect on predictability and availability under a failure.

6. The method of claim 5, wherein the at least one a metric comprises an outage duration metric that correlates a duration of an outage of the at least one operator with a degradation in quality of an output of the application.

7. The method of claim 5, wherein the at least one metric comprises a data dependency level metric that defines how the fault affects a quality of an output of the application when injected at different stream offsets.

8. The method of claim 5, wherein the at least one a metric comprises a quality impact metric that defines a deviation of a quality of an output of the application over local sections of the output.

9. The method of claim 5, wherein the at least one a metric comprises and a recovery time metric that estimates how long it takes the application to recover after occurrence of the fault.

10. The method of claim 1, wherein the fault is injected at different stream offsets and with different outage durations.

11. An article of manufacture, comprising:

a computer readable storage medium containing an executable program for implementing partial fault tolerance in a stream processing application comprising a plurality of stream operators, where the program performs steps comprising;

defining a quality score function that expresses how well the application is performing quantitatively;

injecting a fault into at least one of the plurality of operators;

assessing an impact of the fault on the quality score function; and

selecting at least one partial fault-tolerant technique for implementation in the application based on the assessing.

12. The article of manufacture of claim 11, wherein the injecting comprises:

modifying source code for the application to produce at least one instance of modified source code, where the at least one instance of modified source code incorporates a fault injection parameter for the at least one partial fault tolerance technique; and

compiling the at least one instance of modified source code into executable code.

13. The article of manufacture of claim 11, wherein the assessing comprises:

calculating the quality score function using an output of the application in a fault-free condition to generate a target quality score function;

calculating the quality score function using an output of the application including the fault to generate a faulty quality score function; and

determining whether a difference between the faulty quality score function and the target quality score function is within a tolerance threshold.

14. The article of manufacture of claim 13, wherein the faulty quality score function is calculated over local sections of the output of the application including the fault.

15. The article of manufacture of claim 11, wherein the quality score function accounts for at least one a metric specific to the at least one of the plurality of operators, and wherein the at least one metric characterizes the application in terms of effect on predictability and availability under a failure.

16. The article of manufacture of claim 15, wherein the at least one a metric comprises an outage duration metric that correlates a duration of an outage of the at least one operator with a degradation in quality of an output of the application.

17. The article of manufacture of claim 15, wherein the at least one metric comprises a data dependency level metric that defines how the fault affects a quality of an output of the application when injected at different stream offsets.

18. The article of manufacture of claim 15, wherein the at least one a metric comprises a quality impact metric that defines a deviation of a quality of an output of the application over local sections of the output.

19. The article of manufacture of claim 15, wherein the at least one a metric comprises and a recovery time metric that estimates how long it takes the application to recover after occurrence of the fault.

20. The article of manufacture of claim 11, wherein the fault is injected at different stream offsets and with different outage durations.

21. A system for implementing partial fault tolerance in a stream processing application comprising a plurality of stream operators, the system comprising:

a pre-processor that receives as inputs;

source code for the application, a target stream operator that is one of the plurality of stream operators and at which a fault is to be injected, a fault model corresponding to a partial fault tolerance technique, and a plurality of injection parameters for the partial fault tolerance technique, wherein the pre-processor produces as output a plurality of instances of modified source code for the application, each of the plurality of instances of modified source code incorporating a different one of the plurality of injection parameters; and

an application compiler coupled to the output of the pre-processor, where the application compiler produces as output executable code for each of the plurality of instances of modified source code.

22. The system of claim 21, further comprising:

a testing module coupled to the pre-processor of the application compiler, where the testing module receives as input the executable code for each of the plurality of instances of modified source code and produces as output an assessment of an adequacy of the partial fault tolerance technique for the application.

23. A method for injecting a fault into a stream operator in a data stream processing application, the method comprising:

developing a fault injection operator having specialized logic to emulate an effect of a failure, the fault injection operator corresponding to a fault model;

embedding the fault injection operator into a data flow graph for the data stream processing application, so as to emulate at least one possible effect of the fault model on the data stream processing application,

wherein the fault injection operator is only embedded only in one or more positions of the data flow graph at which the fault model manifests, and

wherein the fault injection operator emulates a different failure when embedded at each of the one or more positions in the data flow graph.

24. The method of claim 23, wherein the fault injection operator is a bursty tuple loss fault injection operator.

25. The method of claim 24, wherein the embedding comprises:

embedding the fault injection operator immediately after each source operator in the data flow graph;

embedding the fault injection operator immediately before each operator in the data flow graph that contains an upstream operator that produces more than one output stream;

embedding the fault injection operator immediately before each operator in the data flow graph that consumes more than one input stream; and

embedding the fault injection operator immediately before each operator in the data flow graph that has a stateful upstream operator.