System for live-migration and automated recovery of applications in a distributed system

• US 11,250,024 B2
• Filed: 09/24/2012
• Issued: 02/15/2022
• Est. Priority Date: 09/23/2011
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Open Invention Network LLC
• Inventors: Marsden, Luke
• Primary Examiner(s): Robinson, Greta L
• Application Number:
  US14/124,221
• Publication Number:
  US 20140129521A1
• Time in Patent Office: 3,431 Days
• Field of Search: 707615, 707616, 707623, 707633, 707634, 707635, 707637, 707638, 707639, 707674, 707686, 707E1701, 709208, 709209, 709211, 709214, 709223, 714 1, 714 12, 718105
• CPC Class Codes: G06F 16/273 Asynchronous replication or reconciliation
• CPC Class Codes: H04L 69/40 for recovering from a failure of a protocol instance or entity, e.g. service redundancy protocols, protocol state redundancy or protocol service redirection management of faults, events, alarms or notifications in data switching networks H04L41/06
• CPC Class Codes: H04W 4/02 Services making use of location information
• CPC Class Codes: H04W 4/90 Services for handling of emergency or hazardous situations, e.g. earthquake and tsunami warning systems [ETWS]

Abstract

A method and apparatus for distribution of applications amongst a number of servers, ensuring that changes to application data on a master for that application are asynchronously replicated to a number of slaves for that application. Servers may be located in geographically diverse locations; the invention permits data replication over high-latency and lossy network connections and failure-tolerance under hardware and network failure conditions. Access to applications is mediated by a distributed protocol handler which allows any request for any application to be addressed to any server, and which, when working in tandem with the replication system, pauses connections momentarily to allow seamless, consistent live-migration of applications and their state between servers. Additionally, a system which controls the aforementioned live-migration based on dynamic measurement of load generated by each application and the topological preferences of each application, in order to automatically keep servers at an optimum utilisation level.

Claims

1. A server configured to deliver a service to at least one client connected to the server, the server operable in either of a master role or a slave role for each of a plurality of applications comprising:

a network interface for connecting the server to at least one other similar server in a cluster of similar servers;

service delivery logic operable when the server is in a master role for an application for hosting that application to deliver the service to the client;

master logic operable when the server is in a master role for an application configured to perform one or more of replicating changes in data of that application to a configurable number of servers of the cluster, and taking snapshots of a file system serving that application and replicating said snapshots to the configurable number of servers in the cluster;

slave logic operable when the server is in a slave role for an application hosted on another server in the cluster configured to receive replicated changes in data from a current master server of the cluster for that application and to maintain a version of said data for that application, and a file system mount handler configured to receive filesystem snapshots from the current master server for that application;

control logic configured to detect an event in the cluster and responsive to the event to autonomously switch the role of the server for one or more of the applications between slave and master, wherein a change of role from slave to master uses the maintained version to host the application and mounts a filesystem for that application using the maintained version of said data for that application or a received filesystem snapshot.

2. A server according to claim 1 wherein the master logic comprises a filesystem mount handler operable in a send mode to transmit the changes in data to the configurable number of servers of the cluster.

3. A server according to claim 2 wherein the master logic comprises a snapshot replicator configured to take snapshots of a filesystem serving a currently hosted application.

4. A server according to claim 2 wherein the master logic comprises at least one per slave sender for replicating changes in data of a live application hosted by the server to a respective server of the cluster.

5. A server according to claim 4 wherein said at least one per slave sender is instantiated by the snapshot replicator for each slave based on the required numbers of slave servers.

6. A server according to claim 1 wherein the slave logic comprises a receive replicator configured to receive the replicated changes in data and a filesystem mount handler configured in receive mode to maintain a version of the live application data.

7. A server according to claim 1 wherein the control logic is configured to emit periodic heart beat signals indicating its live presence in the cluster.

8. A server according to claim 1 wherein the control logic is configured to receive heart beat signals from other similar servers in the cluster, and thereby determine the live presence status of servers in the cluster.

9. A server according to claim 1 wherein the control logic is configured to detect an event selected from the following:

(i) failure of current master server for an application;

(ii) a partitioning of the cluster;

(iii) a reduction in the number of servers in the cluster;

(iv) an increase in the number of servers in the cluster;

(v) introduction into the cluster of a server for which a user has expressed a preference or hosting an application;

(vi) a change in load of applications amongst servers in the cluster, such that a load rebalancing event is required.

10. A server according to claim 1 wherein the control logic is configured to send and receive messages from other servers in the cluster, said messages conveying data whereby an autonomous decision can be made about the role of the server for an application.

11. A server according to claim 10 wherein said messages include binary data indicating said autonomous decision.

12. A server according to claim 1 wherein the control logic is configured to detect messages from all live present servers in the cluster, and to receive messages from all such servers prior to making a decision about its role for an application.

13. A server according to claim 1 wherein the network interface is operable to maintain permanent connections to the least one other similar server in a cluster whereby messages between the servers can be exchanged.

14. A server according to claim 1 wherein the or another network interface is configured to establish temporary sessions for the transmission of the replicated changes in data.

15. A server according to claim 1 comprising a protocol handler operable to route requests for the service to the server when that server is hosting the live application.

16. A server according to claim 1 when configured to act as a master server for a first live application and as a slave server for a second live application hosted on a different server.

17. A system comprising a plurality of servers configured to deliver a service to at least one client connected to the plurality of servers, each server operable in either a master role or a slave role for each of a plurality of applications, each server comprising:

a respective network interface for connecting the respective server to at least one other server of the plurality of servers;

respective service delivery logic operable when the respective server is in a master role for an application for hosting that application to deliver the service to the client;

respective master logic operable when the respective server is in a master role for an application configured to perform one or more of replicating changes in data of that application to a configurable number of other servers of the plurality of servers, and taking snapshots of a file system serving that application and replicating said snapshots to the configurable number of servers in the cluster;

respective slave logic operable when the respective server is in a slave role for an application hosted on another server of the plurality of servers configured to receive replicated changes in data from a current master server of the plurality of servers for that application and to maintain a respective version of said data for that application, and a file system mount handler configured to receive filesystem snapshots from the current master server for that application;

respective control logic configured to detect an event in the plurality of servers and responsive to the event to autonomously switch the role of the respective server for one or more of the applications between slave and master, wherein a change of role from slave to master uses the respective maintained version to host the application and mounts a filesystem for that application using the maintained version of said data for that application or a received filesystem snapshot.

18. Computer software which, when executed at a server operable in either of a master role or a slave role for each of a plurality of applications and comprising a network interface for connecting the server to at least one other similar server in a cluster of similar servers, causes the server to implement:

service delivery logic operable when the server is in a master role for an application for hosting that application to deliver the service to the client;

master logic operable when the server is in a master role for an application configured to perform one or more of replicating changes in data of that application to a configurable number of servers of the cluster, and taking snapshots of a file system serving that application and replicating said snapshots to the configurable number of servers in the cluster;

slave logic operable when the server is in a slave role for an application hosted on another server in the cluster configured to receive replicated changes in data from a current master server of the cluster for that application and to maintain a version of said data for that application, and a file system mount handler configured to receive filesystem snapshots from the current master server for that application;

control logic configured to detect an event in the cluster and responsive to the event to autonomously switch the role of the server for one or more of the applications between slave and master, wherein a change of role from slave to master uses the maintained version to host the application and mounts a filesystem for that application using the maintained version of said data for that application or a received filesystem snapshot.