Mechanism for Recovery from Site Failure in a Stream Processing System

• US 20080256384A1
• Filed: 04/10/2007
• Published: 10/16/2008
• Est. Priority Date: 04/10/2007
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Liu, Zhen, Branson, Michael John, Douglis, Frederick, Fawcett, Bradley William, Rong, Bin
• Granted Patent: US 8,219,848 B2
• Time in Patent Office: Days
• US Class Current: 714/4
• CPC Class Codes: G06F 11/2025 using centralised failover control functionality
• CPC Class Codes: G06F 11/2048 where the redundant components share neither address space nor persistent storage
• CPC Class Codes: G06F 11/2097 maintaining the standby controller/processing unit updated initialisation or re-synchronisation thereof G06F11/1658 and subgroups

Abstract

A failure recovery framework to be used in cooperative data stream processing is provided that can be used in a large-scale stream data analysis environment. Failure recovery supports a plurality of independent distributed sites, each having its own local administration and goals. The distributed sites cooperate in an inter-site back-up mechanism to provide for system recovery from a variety of failures within the system. Failure recovery is both automatic and timely through cooperation among sites. Back-up sites associated with a given primary site are identified. These sites are used to identify failures within the primary site including failures of applications running on the nodes of the primary site. The failed applications are reinstated on one or more nodes within the back-up sites using job management instances local to the back-up sites in combination with previously stored state information and data values for the failed applications. In additions to inter-site mechanisms, each one of the plurality of sites employs an intra-site back-up mechanism to handle failure recoveries within the site.

Claims

1. A method for providing failure recovery in cooperative data stream processing, the method comprising:

identifying a plurality of distributed sites, each site comprising one or more nodes and capable of independently hosting on the nodes applications associated with jobs derived from inquiries to process continuous dynamic streams of data; and

using an inter-site back-up mechanism to provide failure recovery, the inter-site back-up mechanism comprising back-up sites selected from the identified plurality of sites.

2. The method of claim 1, wherein the step of using the inter-site back-up mechanism further comprises:

identifying a primary site from the plurality of distributed sites; and

selecting a plurality of back-up sites from the plurality of distributed sites to provide failure recovery for the identified primary site.

3. The method of claim 1, wherein the step of using the inter-site back-up mechanism further comprises saving current state information and current data values at pre-determined intervals for one or more applications hosted on the nodes.

4. The method of claim 3, wherein the step of saving current state information and current data values further comprises:

identifying high priority applications having associated priority levels above a pre-determined level to have state information and data values saved; and

identifying for each identified application the pre-determined interval for saving state information and data values.

5. The method of claim 4, wherein the step of identifying the pre-determined interval further comprises identifying the pre-determined interval empirically.

6. The method of claim 2, wherein the step of using the inter-site back-up mechanism further comprises identifying a monitoring site to monitor for a periodically emitted signal from the identified primary site indicating that the primary site is operating within prescribed parameters.

7. The method of claim 6, wherein the step of using the identified monitoring site to monitor for the periodically emitted signal further comprises:

identifying a pre-defined interval for sending the periodically emitted signal from the identified primary site to the identified monitoring site; and

initiating an inter-site failure recovery procedure if the periodically emitted signal is not received at the monitoring site within the pre-defined interval.

8. The method of claim 6, wherein the step of using the inter-site back-up mechanism further comprises:

identifying a plurality of additional sites to monitor for the periodically emitted signal;

communicating status of receipt of the periodically emitted signals at each one of the additional sites to the monitoring site;

identifying a pre-defined interval for sending the periodically emitted signal from the identified primary site to the identified monitoring site; and

initiating an inter-site failure recovery procedure if the periodically emitted signal is not received at the monitoring site within the pre-defined interval and the status of receipt at all of the additional monitoring sites is that the periodically signal was not received.

9. The method of claim 7, wherein the step of initiating the inter-site failure recovery procedure further comprises using a failover manager in communication with each one of the identified back-up sites to prohibit more than one of the identified back-up sites from initiating the inter-site failure recovery.

10. The method of claim 6, wherein:

the method further comprises identifying one or more critical applications running on the primary site; and

the step of using the inter-site back-up mechanism further comprises;

identifying a failure condition on the primary site; and

recovering the identified critical applications for the failed primary site.

11. The method of claim 10, wherein the step of recovering the critical applications comprises retrieving previously saved state information and data values for the critical applications, using job manager instances on the back-up sites to reinstate the critical applications, reestablishing connections between the critical and appropriate data streams, reestablishing tunneling among sites for inter-site output of data generated by the reinstated applications or combinations thereof.

12. The method of claim 10, wherein the step of recovering the critical applications comprises:

halting existing tunneling with the failed primary site;

creating new monitoring agreements to handle the recovered critical applications;

reconfiguring tunneling with the recovered critical applications; and

re-deploying critical applications to back-up sites.

13. The method of claim 10, wherein the step of using the inter-site back-up mechanism further comprises notifying sites cooperating with the failed primary site and affected by the failure of the primary site.

14. The method of claim 10, wherein the step of reinstating the critical applications further comprises compensating for differences in execution environments between the primary site and the back-up sites.

15. The method of claim 1, further comprising using an intra-site back-up mechanism on each one of the plurality of identified sites to provide failure recovery.

16. The method of claim 15, wherein the step of using an intra-site back-up mechanism further comprises:

terminating lower priority applications running on nodes within a given site; and

redistributing higher priority applications among all of the nodes on the given site.

17. The method of claim 14, wherein:

each site comprises an inquiry services layer to produce jobs from a high level description of end results and to facilitate description of jobs in a desired language; and

the step of using the intra-site back-up mechanism further comprises using the inquiry services layer to replan, reproduce and execute at least a portion of the jobs executing on a given site based upon resource availability on that site following a failure of the primary site.

18. The method of claim 1, wherein each site comprising components capable of independently processing continuous dynamic streams of data;

and

The step of using the inter-site back-up mechanism further comprises using the inter-site back-up mechanism to recover the components running on each site.

19. A computer-readable medium containing a computer-readable code that when read by a computer causes the computer to perform a method for providing failure recovery in cooperative data stream processing, the method comprising:

identifying a plurality of distributed sites, each site comprising one or more nodes and capable of independently hosting on the nodes applications associated with jobs derived from inquiries to process continuous dynamic streams of data; and

using an inter-site back-up mechanism to provide failure recovery, the inter-site back-up mechanism comprising back-up sites selected from the identified plurality of sites.

20. The computer-readable medium of claim 19, wherein the step of using the inter-site back-up mechanism further comprises:

identifying a primary site from the plurality of distributed sites; and

selecting a plurality of back-up sites from the plurality of distributed sites to provide failure recovery for the identified primary site.