System for live-migration and automated recovery of applications in a distributed system

• US 9,477,739 B2
• Filed: 12/17/2013
• Issued: 10/25/2016
• Est. Priority Date: 09/23/2011
• Status: Expired due to Fees

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Hybrid Logic Ltd.
• Inventors: Marsden, Luke
• Primary Examiner(s): ROBINSON, GRETA LEE
• Application Number:
  US14/109,434
• Publication Number:
  US 20140108342A1
• Time in Patent Office: 1,043 Days
• Field of Search: 707/615, 707/616, 707/623, 707/633, 707/634, 707/635, 707/637, 707/638, 707/639, 707/674, 707/686, 707/E17.01, 714/1, 714/2, 714/6.12, 714/6.3, 718/105
• US Class Current: 1/1
• CPC Class Codes: G06F 16/273 Asynchronous replication or reconciliation

Abstract

A method and apparatus for distribution of applications amongst a number of servers, ensuring that changes to application data on a master for that application are asynchronously replicated to a number of slaves for that application. Servers may be located in geographically diverse locations; the invention permits data replication over high-latency and lossy network connections and failure-tolerance under hardware and network failure conditions. Access to applications is mediated by a distributed protocol handler which allows any request for any application to be addressed to any server, and which, when working in tandem with the replication system, pauses connections momentarily to allow seamless, consistent live-migration of applications and their state between servers. Additionally, a system which controls the aforementioned live-migration based on dynamic measurement of load generated by each application and the topological preferences of each application, in order to automatically keep servers at an optimum utilization level.

Claims

1. A method of managing a plurality of applications hosted by a cluster of servers which each have an interface connectable to at least one client by a network, each application delivering a service at the client, the method comprising:

electing a server of the cluster as a master server, the master server hosting at least one live application; and

while the master server is hosting the live application, replicating changes in application data of the live application to a configurable number of servers in the cluster elected as slave servers whereby each elected slave server maintains a version of the application data of the live application, wherein responsive to an event in the cluster hosting of the application is transferred from the master server to one of the elected slave servers determined without intervention by a user when the event is detected, the elected slave server using its version of the current application data, to mount the application and become a new master server, and wherein said event is at least one of;

detection of a preferred alternate master server in the cluster based on the loads of servers in the cluster;

detection of a preferred alternate master server based on the locality of servers in the cluster;

detection of a preferred alternate master server in the cluster based on a predefined user preference;

detected by exchanging messages with other servers of the cluster;

addition of a server to the cluster;

removal of a server from the cluster, wherein the removal was anticipated;

failure of a server in the cluster, wherein the data for live applications hosted by the failed server is recovered using versions of the current application on servers in the cluster which are continuing to operate; and

a partition of the cluster, wherein after recovery from the partition a preferred alternate master server is selected from a number of potentially competing master servers as the server with the version of the current application data which is more valuable.

2. A method according to claim 1 wherein a decision to migrate a live application from its current master server to one of its slave servers is made when the load of the current master is greater than an average value of loads of all of the servers in the cluster and a damping factor (Q).

3. A method according to claim 1 wherein the cluster included a single server prior to addition of the one server.

4. A method according to claim 1 wherein on addition of a server to the cluster, the new load of the servers is determined, and a decision as to which of the applications hosted by the cluster should be migrated to the newly added server is made.

5. A method according to claim 1 wherein a leader function hosted on one of the servers determines the new master server for the application, wherein the leader function is on a different server from the master.

6. A method according to claim 1 wherein a leader function hosted on the current master server determines the new master server.

7. Computer software which, when executed by appropriate processing means, causes the processing means to implement a system for replicating a filesystem between a first server and a second server prior to and following a partition between the first server and the second server, the system comprising:

snapshotting means for taking snapshots of a current state of the filesystem on the first server at predetermined points in time following modification of the filesystem, each snapshot recording differences between the current state of the filesystem on the server and a state of the filesystem on the server at the time point of a previous snapshot;

replicator means for continually replicating the snapshots taken on the first server to the second server as soon as they are taken;

detection means configured such that upon detection of a partition, both the first and the second server become masters for the filesystem and accept new modifications to the filesystems; and

updating means configured to perform an update process to update the filesystem after recovery of the partition, the update process comprising;

identifying which of the first server and the second server contains the most current version of the filesystem;

nominating the server so identified as containing the most current version of the file system to be the master server and the other server as the slave server;

identifying a snapshot that is common to both the master server and the slave server; and

replicating subsequent snapshots from the master server to the slave server.

8. Computer software according to claim 7 wherein identifying which of the first server and the second server contains the most current version of the filesystem comprises calculating a centre of mass metric for the version of the filesystem on each of the servers, the centre of mass metric representing the average age of the snapshots of the filesystem on each server and the number of changes to the filesystem represented by the snapshots on each server.

9. Computer software according to claim 7 wherein identifying which of the first server and the second server contains the most current version of the filesystem further comprises identifying a set of snapshots of the filesystem that for each server, each set of snapshots containing snapshots only present on that server, and calculating the centre of mass metric for each server based on that server'"'"'s set of snapshots.

10. Computer software according to claim 7 wherein the update process further comprises storing the snapshots of the slave server that were taken after the common snapshot.

11. Computer software according to claim 7 configured to store the snapshots taken of the filesystem such that a previous snapshot of the filesystem can be selected by a user from the stored snapshots to restore the system to its state at the time of the selected snapshot.