System and method for event-driven live migration of multi-process applications

• US 9,043,640 B1
• Filed: 12/01/2010
• Issued: 05/26/2015
• Est. Priority Date: 08/26/2005
• Status: Expired due to Fees

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Open Invention Network LLC
• Inventors: Havemose, Allan
• Primary Examiner(s): TRUONG, LOAN
• Application Number:
  US12/957,637
• Time in Patent Office: 1,637 Days
• Field of Search: 714/6.3, 714/4.12, 714/32, 714/15, 714/13, 707/610, 719/319
• US Class Current: 714/15
• CPC Class Codes: G06F 11/1438 Restarting or rejuvenating
• CPC Class Codes: G06F 11/1451 by selection of backup contents
• CPC Class Codes: G06F 11/1464 for networked environments
• CPC Class Codes: G06F 11/1469 Backup restoration techniques
• CPC Class Codes: G06F 11/1471 involving logging of persistent data for recovery
• CPC Class Codes: G06F 11/1482 by means of middleware or OS functionality
• CPC Class Codes: G06F 11/1497 Details of time redundant execution on a single processing unit
• CPC Class Codes: G06F 11/1683 at instruction level
• CPC Class Codes: G06F 11/203 using migration
• CPC Class Codes: G06F 11/2041 with more than one idle spare processing component
• CPC Class Codes: G06F 11/2046 where the redundant components share persistent storage G06F11/2043 takes precedence
• CPC Class Codes: G06F 11/2097 maintaining the standby controller/processing unit updated initialisation or re-synchronisation thereof G06F11/1658 and subgroups
• CPC Class Codes: G06F 2201/805 Real-time
• CPC Class Codes: G06F 2201/82 Solving problems relating to consistency ensuring consistency in mirrored systems G06F11/2064
• CPC Class Codes: G06F 2201/825 the problem or solution involving locking
• CPC Class Codes: G06F 2201/84 Using snapshots, i.e. a logical point-in-time copy of the data

Abstract

A system, method, and computer readable medium for asynchronous live migration of applications between two or more servers. The computer readable medium includes computer-executable instructions for execution by a processing system. Primary applications runs on primary hosts and one or more replicated instances of each primary application run on one or more backup hosts. Asynchronous live migration is provided through a combination of process replication, logging, barrier synchronization, checkpointing, reliable messaging and message playback. The live migration is transparent to the application and requires no modification to the application, operating system, networking stack or libraries.

Claims

1. A system for providing live migration of a primary application to one or more backup applications, the system comprising:

one or more computer system memory locations configured to store said primary application;

one or more Central Processing Units (CPUs) operatively connected to said computer system memory and configured to execute said primary application on a primary host with a host operating system;

one or more interceptors configured to intercept calls to the host operating system and shared libraries, and configured to generate replication messages based on said intercepted calls, wherein said interceptors intercept at least one or more of process operations, thread operations, file operations, lock operations, Input operations/Output operations, and resource operations;

a messaging layer for said primary application configured to transmit said replication messages to the one or more backups;

a logging facility for said messaging layer configured to log all replication messages and checkpoints;

a checkpointing service for said primary application configured to checkpoint said primary application; and

one or more backup hosts each with a host operating system and each comprising;

computer system memory comprising one or more computer system memory locations configured to store one or more backup applications, and one or more Central Processing Units (CPUs) operatively connected to said computer system memory and configured to execute said one or more backup applications;

one or more interceptors configured to intercept calls to said one or more backup host operating systems and shared libraries;

a messaging layer for each one or more backup applications configured to provide ordered receipt of said replication messages; and

one or more checkpointing services for said one or more backup applications;

wherein live migration is performed in response to an event;

wherein a replication message optionally contains a DATA block in which the interceptor for the operation for said replication message stores information required during message replay comprised of at least one of a return value, results, parameters, or state for said operation, or no DATA block if no such information is required;

wherein restore on a backup is initiated by loading the most recent checkpoint from the logging facility;

wherein replication messages subsequent to said most recent checkpoint are read from the logging facility and replayed to the backup; and

wherein said replayed replication messages subsequent to said most recent checkpoint uses information stored in a DATA block instead of executing the operations associated with the replication message if said DATA block is available including using at least one of a return value, results, parameters, or a state generated by the primary application and transmitted using a DATA block instead of executing the operation on the backup, or executes the operation if said DATA block is not available.

2. The system according to claim 1, where said operating system is one of Linux, UNIX or Windows.

3. The system according to claim 1, wherein the messaging layers are configured to transmit said messages over one of User Datagram Protocol (UDP), Transmission Control Protocol (TCP), UDP using multicast, or UDP using broadcast.

4. The system according to claim 1, wherein said event is a fault, and said fault is one of application crash, host crash, operating system fault, memory fault, storage fault, power supply fault, general device fault, Central Processing Unit (CPU) threshold fault, memory threshold fault, storage threshold fault, or general script generated fault.

5. The system according to claim 1, wherein the messaging layers on the one or more backups are configured to compare the last processed replication message prior to the fault to the replication messages in the logging facility, and is configured to use the results from the primary as stored in the DATA block.

6. The system according to claim 1, wherein the messaging layer on the one or more backups are configured to replay replication messages in the logging facility not previously received by the backup.

7. The system according to claim 1, wherein the system is configured to choose a backup host based on one of preconfigured backup, operator-chosen backup, or dynamically chosen backup based on available resources.

8. The system according to claim 1, wherein the system is configured to promote a backup to primary upon replaying the last replication message from the logging facility.

9. The system according to claim 8, wherein said backup is configured to switch interceptors to primary interceptors as part of said promotion to primary.

10. A system for restoring a primary application to a backup application on a backup host, the system comprising:

one or more computer system memory locations configured to store said primary application;

one or more Central Processing Units (CPUs) operatively connected to said computer system memory and configured to execute said primary application on a primary host with a host operating system;

one or more interceptors configured to intercept calls to the host operating system and shared libraries, and configured to generate replication messages based on said intercepted calls, wherein said interceptors intercept at least one or more of process operations, thread operations, file operations, lock operations, Input operations/Output operations, and resource operations;

a messaging layer for said primary application configured to transmit said replication messages to the one or more backups;

a logging facility for said messaging layer configured to log all replication messages and checkpoints;

a checkpointing service for said primary application configured to checkpoint said primary application; and

one or more backup hosts each with a host operating system and each comprising;

computer system memory comprising one or more computer system memory locations configured to store one or more backup applications, and one or more Central Processing Units (CPUs) operatively connected to said computer system memory and configured to execute said one or more backup applications;

one or more available interceptors configured to intercept calls to the backup host operating systems and shared libraries;

an available messaging layer for the backup application configured to provide ordered receipt of said replication messages; and

an available checkpointing services for said backup application;

wherein a replication message optionally contains a DATA block in which the interceptor for the operation for said replication message stores information required during message replay comprised of at least one of a return value, results, parameters, or state for said operation, or no DATA block if no such information is required;

wherein restore on the backup is configured to load said primary application on the backup host, configured to restore the application state from the most recent checkpoint stored in the logging facility, configured to load said one or more interceptors, configured to load said checkpointing service, configured to load said messaging layer, configured to read replication messages from the logging facility subsequent to said most recent checkpoint, and configured to replay said messages subsequent to said most recent checkpoint using information stored in a DATA block instead of executing the operations associated with the replication message if said DATA block is available including using at least one of a return value, results, parameters, or state generated by the primary application and transmitted using a DATA block instead of executing the operation on the backup, or executing the operation if said DATA block is not available.

11. The system according to claim 10, wherein said operating system is one of Linux, UNIX or Windows.

12. The system according to claim 10, wherein the messaging layer is configured to transmit said messages over one of User Datagram Protocol (UDP), Transmission Control Protocol (TCP), UDP using multicast, or UDP using broadcast.

13. The system according to claim 10, where said event is a fault, and said fault is one of application crash, host crash, operating system fault, memory fault, storage fault, power supply fault, general device fault, Central Processing Unit (CPU) threshold fault, memory threshold fault, storage threshold fault, or general script generated fault.

14. The system according to claim 10, wherein said backup application is configured to promote to primary upon processing of the last replication message from the logging facility.

15. The system according to claim 10, further configured to boot a backup server if a backup server is not available.

16. The system according to claim 10, wherein the system is configured to choose the backup based on one of preconfigured backup, operator-chosen backup, or dynamically chosen backup based on available resources.

17. The system according to claim 10, wherein the backup application is configured to switch to new primary interceptors from backup interceptors as part of the promotion to primary.

18. The system according to claim 10, wherein said restore is performed in response to an event.

19. The system according to claim 18, wherein said event is one of application crash, host crash, operating system fault, memory fault, storage fault, power supply fault, general device fault, Central Processing Unit (CPU) threshold fault, memory threshold fault, storage threshold fault, script generated fault, Simple Network Management Protocol (SNMP) event, or operator generated event.

20. The system according to claim 10, wherein the backup application is configured to promote said restored application to primary.