Method and system for providing high availability to computer applications

• US 9,311,200 B1
• Filed: 01/23/2012
• Issued: 04/12/2016
• Est. Priority Date: 08/26/2004
• Status: Active Grant

• Current Assignee: Red Hat, Inc. (International Business Machines Corporation)
• Original Assignee: Red Hat, Inc. (International Business Machines Corporation)
• Inventors: Ngan, Ching-Yuk Paul, Hortikar, Sanjay D., Havemose, Allan, Tewksbury, Lauren, Raman, Vishwas
• Primary Examiner(s): Leibovich, Yair
• Application Number:
  US13/356,223
• Time in Patent Office: 1,541 Days
• Field of Search: 714/10, 714 1- 3, 714/4.1, 714/4.11, 714/4.12
• US Class Current: 1/1
• CPC Class Codes: G06F 11/0709 in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
• CPC Class Codes: G06F 11/1402 Saving, restoring, recovering or retrying
• CPC Class Codes: G06F 11/141 for bus or memory accesses
• CPC Class Codes: G06F 11/1438 Restarting or rejuvenating
• CPC Class Codes: G06F 11/1482 by means of middleware or OS functionality
• CPC Class Codes: G06F 11/20 using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
• CPC Class Codes: G06F 11/2002 where interconnections or communication control functionality are redundant flexible arrangements for bus networks involving redundancy H04L12/40176
• CPC Class Codes: G06F 11/2023 Failover techniques
• CPC Class Codes: G06F 11/2028 eliminating a faulty processor or activating a spare
• CPC Class Codes: G06F 11/203 using migration
• CPC Class Codes: G06F 11/2046 where the redundant components share persistent storage G06F11/2043 takes precedence
• CPC Class Codes: G06F 11/30 Monitoring
• CPC Class Codes: H04L 61/5007 Internet protocol [IP] addresses

Abstract

A set of system-level high availability services for computer systems, including a service that functions in general terms like an extension of the operating system. By providing High Availability (HA) at the system-level, modifications to the applications or the operating system kernel are not required.

Claims

1. A method, comprising:

preloading a high-availability shared library with a custom function using an operating system loader, wherein the high-availability shared library is provided within a set of system-level services, and the custom function initializes high-availability services for an application program before the application program is loaded on a primary node;

registering, by the custom function of the high-availability shared library, the application program with a high availability protection program, wherein the high-availability shared library is loaded prior to registering the application program;

after the registering, loading the application program on the primary node; and

assigning a virtual IP address for the application program running on the primary nod,

wherein modification of the application program or application program recompilation is not required.

2. A method as recited in claim 1, comprising:

monitoring unregistration of said application program from said high availability program; and

designating said unregistration as a normal exit.

3. A method as recited in claim 1, wherein the virtual IP address is assigned by the high-availability shared library to a plurality of application programs running on the primary node, the method further comprising:

migrating each application program of the plurality of application programs to a backup location in response to an application program of the plurality of application programs failing over to the backup location, wherein the virtual IP address is unique, mapped to a single network interface card (NIC), and associated with the plurality of application programs at the backup location.

4. A method as recited in claim 1, the method comprising:

assigning the virtual IP address for a second application program running on the primary node, the second application program being different from the first application program;

when the first application program for which the virtual IP address is assigned fails over to a backup node, moving the second application program for which the virtual IP address is assigned to the backup node based on the first application program failover to the backup node.

5. A method as recited in claim 1, wherein the initialization and registration occurs before any part of the application has been loaded.

6. A non-transitory machine-readable medium comprising a plurality of machine-readable instructions that when executed by one or more processors is adapted to cause the one or more processors to perform a method comprising:

preloading a high-availability shared library with a custom function using an operating system loader, wherein the high-availability shared library is provided within a set of system-level services, the custom function initializes high-availability services for an application program, and the high-availability shared library is loaded before the application program is loaded on a node;

registering, by the custom function of the high-availability shared library, the application program with a high availability protection program;

after the registering, loading the application program on the node; and

assigning a virtual IP address for the application program running on the node,

wherein modification of the application program or application program recompilation is not required.

7. A machine-readable medium as recited in claim 6, comprising:

monitoring unregistration of said application program from said high availability program; and

designating said unregistration as a normal exit.

8. A machine-readable medium as recited in claim 6, wherein registration code comprises one or more of:

registering the application program with high availability services, initializing check pointing, initializing fault detection, initializing virtual IP addresses, initializing a kernel module, initializing recovery services, initializing other high availability services, and registering an exit handler.

9. A non-transitory machine-readable medium comprising a plurality of machine-readable instructions that when executed by one or more processors is adapted to cause the one or more processors to perform a method comprising:

preloading a shared library with a custom function using an operating system loader;

after preloading the shared library, loading an application program on a node including a network interface;

registering, by the custom function, the application program with a high availability protection program, wherein the shared library is loaded prior to loading the application program; and

assigning, by the shared library, a virtual IP address to a plurality of application programs running on the node, the virtual IP address being mapped to the network interface of the node, and the plurality of application programs including the application program;

migrating each application program of the plurality of application programs to a backup location in response to an application program of the plurality of application programs failing over to the backup location, wherein the virtual IP address is unique and assigned to the plurality of application programs at the backup location;

wherein modification of the application program or application program recompilation is not required.

10. A machine-readable medium as recited in claim 9, wherein said program performs the steps comprising:

monitoring unregistration of said application program from said high availability program; and

designating said unregistration as a normal exit.

11. A machine-readable medium method as recited in claim 9, wherein the high-availability shared library is provided within a set of system-level services, and the assignment of the virtual IP address upon failover to the backup server is provided as a system-level function of the set of system-level services.

12. A method, comprising:

providing a loadable kernel module;

providing an application checkpointing procedure which invokes said kernel module;

providing an application restoration procedure which invokes said kernel module;

wherein, during checkpointing, the checkpointing procedure notifies the kernel module, and the kernel module captures, by a bridge at an IP layer of a TCP/IP stack of a primary node, a state of all connections at the primary node that are open for that process in a buffer and sends the state to the checkpointing procedure; and

sending the connection state from the primary node to a TCP/IP stack of a backup node along with an application state;

wherein, during restoration on the backup node, the restoration procedure invokes the kernel module with the saved buffer, and the kernel module uses the data in the buffer to restore all the connections in the exact same state as they were in the primary node.

13. A method as recited in claim 12, wherein the kernel module hooks into the networking stack during checkpointing and restoration and freezes the entire connection;

and

wherein the client experiences a connection delay during the time the process is checkpointed and restored on the backup node.

14. A non-transitory machine-readable medium comprising a plurality of machine-readable instructions that when executed by one or more processors is adapted to cause the one or more processors to perform a method comprising:

a loadable kernel module;

an application checkpointing procedure which invokes said kernel module;

an application restoration procedure which invokes said kernel module;

wherein, during checkpointing, the checkpointing procedure notifies the kernel module, and the kernel module captures, by a bridge at an IP layer of a TCP/IP stack of a primary node, a state of all the connections at the primary node that are open for that process in a buffer and sends the state to the checkpointing procedure;

wherein the checkpointing procedure sends the connection state from the primary node to a backup node along with an application state; and

wherein, during restoration on the backup node, the restoration procedure invokes the kernel module with the saved buffer, and the kernel module uses the data in the buffer to restore all the connections in the exact same state as they were in the primary node.

15. A machine-readable medium as recited in claim 14, wherein the kernel module hooks into the networking stack during checkpointing and restoration and freezes the entire connection;

and wherein the client experiences a connection delay during the time the process is checkpointed and restored on the backup node.

16. A method, comprising:

preloading a shared library with a custom function using an operating system loader;

after preloading the shared library, loading an application program on a primary server;

aliasing a first unique IP address of the application program, by the shared library, to an IP address of the primary server;

detecting a failure in an execution of the application program running on the primary server; and

executing said application program from one or more designated backup servers automatically in response to said failure, wherein the application program remains responsive on the IP address of the primary server while executing on one or more of the designated backups servers;

managing high availability policy using an automatic profiler monitoring application execution, or in response to policy adjustments from an administrator, wherein said profiler monitors application execution, builds a statistical description of the execution and uses the statistical description for generating and/or recommending optimal settings;

wherein said function registers said application program with a high availability protection program; and

wherein modification of the application program or application program recompilation is not required.

17. A method as recited in claim 16, comprising, monitoring unregistration of said application program from said high availability program;

and

designating said unregistration as a normal exit.

18. A method as recited in claim 16, wherein said policy comprises characteristics for controlling a heart-beat frequency, a checkpointing interval, configuration of health-checks, or start/stop/restart scripts.

19. A method as recited in claim 16, comprising performing healthchecks during the execution of said application program for said detecting of a failure in the execution of said application program.

20. A method, comprising:

loading a kernel module;

executing a checkpointing procedure for an application to invoke said kernel module for capturing, by a bridge at an IP layer of a TCP/IP stack of a primary server, a state of connections at the primary server that are open for that process and sending the connection state to said checkpointing procedure and from the primary server to a backup server node in conjunction with application state; and

executing application restoration procedures to invoke said kernel module for restoring the backup server node to restore connections in the exact state as they were in the primary server.

21. A method, as recited in claim 20, wherein the kernel module hooks into the networking stack during checkpointing and restoration and freezes the entire connection;

and wherein the client experiences a connection delay during the time the process is checkpointed and restored on the backup node.

22. A non-transitory machine-readable medium comprising a plurality of machine-readable instructions that when executed by one or more processors is adapted to cause the one or more processors to perform a method comprising:

a loadable kernel module;

an application checkpointing procedure configured for invoking said kernel module for capturing, by a bridge at an IP layer of a TCP/IP stack of the primary node, a state of all connections at the primary node for that process and communicating the connection states back to the checkpointing procedure and from the primary node to a TCP/IP stack of a backup node along with the application state; and

an application restoration procedure configured for invoking said kernel module to restore all the connections in the exact same state as they were in the primary node.

23. A machine-readable medium as recited in claim 22, wherein the kernel module hooks into the networking stack during checkpointing and restoration and freezes the entire connection;

and wherein the client experiences a connection delay during the time the process is checkpointed and restored on the backup node.