System for live-migration and automated recovery of applications in a distributed system

• US 10,311,027 B2
• Filed: 12/17/2013
• Issued: 06/04/2019
• Est. Priority Date: 09/23/2011
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Open Invention Network LLC
• Inventors: Marsden, Luke
• Primary Examiner(s): Robinson, Greta L
• Application Number:
  US14/109,545
• Publication Number:
  US 20140108350A1
• Time in Patent Office: 1,995 Days
• Field of Search: 707615, 707616, 707623, 707633, 707634, 707635, 707637, 707638, 707639, 707674, 707686, 707E1701, 707663, 707666, 709208, 709209, 709211, 709214, 709223, 714 1, 714 12, 718105
• CPC Class Codes: G06F 16/184 implemented as replicated file system
• CPC Class Codes: G06F 2209/505 Clust
• CPC Class Codes: G06F 2209/508 Monitor
• CPC Class Codes: G06F 3/061 Improving I/O performance
• CPC Class Codes: G06F 3/0647 Migration mechanisms
• CPC Class Codes: G06F 3/065 Replication mechanisms
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: G06F 9/5027 the resource being a machine, e.g. CPUs, Servers, Terminals
• CPC Class Codes: G06F 9/5061 Partitioning or combining of resources
• CPC Class Codes: H04L 67/01 Protocols
• CPC Class Codes: H04L 67/1095 Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
• CPC Class Codes: H04L 67/148 Migration or transfer of sessions
• CPC Class Codes: H04L 69/40 for recovering from a failure of a protocol instance or entity, e.g. service redundancy protocols, protocol state redundancy or protocol service redirection management of faults, events, alarms or notifications in data switching networks H04L41/06

Abstract

A method and apparatus for distribution of applications amongst a number of servers, ensuring that changes to application data on a master for that application are asynchronously replicated to a number of slaves for that application. Servers may be located in geographically diverse locations; the invention permits data replication over high-latency and lossy network connections and failure-tolerance under hardware and network failure conditions. Access to applications is mediated by a distributed protocol handler which allows any request for any application to be addressed to any server, and which, when working in tandem with the replication system, pauses connections momentarily to allow seamless, consistent live-migration of applications and their state between servers. Additionally, a system which controls the aforementioned live-migration based on dynamic measurement of load generated by each application and the topological preferences of each application, in order to automatically keep servers at an optimum utilization level.

Claims

1. Computer software which, when executed by appropriate processing means, causes the processing means to implement a method of managing snapshots of a filesystem, where the filesystem is replicated across multiple servers connected in a cluster comprising:

indentifying each snapshot by a snapnode object in the form of a binary sequence comprising a snapshot identifier, a parent pointer to an earlier snapshot on a specific server where the snapshot was taken, and the set of servers where this snapshot is presently stored;

storing a graph of snapnode objects of a set of snapshots of a filesystem on each of the multiple servers, one of the servers being an active master of the file system;

the active master taking a new snapshot of the filesystem and creating a snapnode object for the new snapshot identifying the active master as a server where the new snapshot is stored; and

transmitting the new snapshot to the other servers of the multiple servers; and

modifying the snapnode object to identify the other servers as servers where the new snapshot is stored,

wherein the method is used to manage recovery of a file system after an event in which the active master is to confirm or modify its status.

2. Computer software according to claim 1 wherein the event is the partitioning of a cluster of servers in which the active master and the other servers are connected, wherein after recovery from the partition there are at least two candidate master servers, each with a graph of snapnode objects for the filesystem, wherein the graph at each candidate master is traversed to assess its value, and the candidate master with the graph indicating a highest value adopts a role as new master for the filesystem.

3. Computer software according to claim 2 wherein prior to executing the comparison, the snapshot data is globally synchronised across the servers, whereby divergence of a version of the data at each candidate master can be assessed against the globally synchronised snapshot data.

4. Computer software according to claim 3 wherein the event is loss of at least one other server of the multiple servers, which was acting as a slave server to the active master, wherein after a replacement slave has been designated by a master, the master instructs the new slave to replicate a complete copy of the filesystem so that replication can begin from a current point.

5. Computer software according to claim 2, the implemented method comprising the step of saving snapshots from a given slice point in the graph to a local storage area.

6. Computer software according to claim 2, the implemented method comprising the step of pruning a snapshot.

7. Computer software according to claim 2, implemented method comprising the step of determining which action to take in order to resolve a divergence of graphs on multiple servers representing the same filesystem based on:

(1) a current master for the filesystem;

(2) a graph of snapnode objects for that filesystems global state;

(3) a list of current slave servers to that master for that filesystem.

8. Computer software according to claim 1, wherein the method further comprises:

detecting an event selected from the following;

(i) failure of current master server for the application, (ii) a partitioning of the cluster, (iii) a reduction in the number of servers in the cluster, (iv) an increase in the number of servers in the cluster, (v) introduction into the cluster of a server for which a user has expressed a preference for hosting the application, (vi) a change in load of applications amongst servers in the cluster, such that a load rebalancing event is required; and

in response to the event determining a new one of the servers being the active master of the filesystem.

9. A method of managing snapshots of a filesystem, where the filesystem is replicated across multiple servers connected in a cluster comprising:

indentifying each snapshot by a snapnode object in the form of a binary sequence comprising a snapshot identifier, a parent pointer to an earlier snapshot on a specific server where the snapshot was taken, and the set of servers where this snapshot is presently stored;

storing a graph of snapnode objects of a set of snapshots of a filesystem on each of the multiple servers, one of the servers being an active master of the file system;

the active master taking a new snapshot of the filesystem and creating a snapnode object for the new snapshot identifying the active master as a server where the new snapshot is stored; and

transmitting the new snapshot to the other servers of the multiple servers; and

modifying the snapnode object to identify the other servers as servers where the new snapshot is stored,

wherein the method is used to manage recovery of a file system after an event in which the active master is to confirm or modify its status.

10. A method according to claim 9 wherein the event is the partitioning of a cluster of servers in which the active master and the other servers are connected, wherein after recovery from the partition there are at least two candidate master servers, each with a graph of snapnode objects for the filesystem, wherein the graph at each candidate master is traversed to assess its value, and the candidate master with the graph indicating a highest value adopts a role as new master for the filesystem.

11. A method according to claim 10 wherein prior to executing the comparison, the snapshot data is globally synchronised across the servers, whereby divergence of a version of the data at each candidate master can be assessed against the globally synchronised snapshot data.

12. A method according to claim 9 wherein the event is loss of at least one other server of the multiple servers, which was acting as a slave server to the active master, wherein after a replacement slave has been designated by a master, the master instructs the new slave to replicate a complete copy of the filesystem so that replication can begin from a current point.

13. A method according to claim 9 comprising the step of saving snapshots from a given slice point in the graph to a local storage area.

14. A method according to claim 9 comprising the step of pruning a snapshot.

15. A method according to claim 9 comprising the step of determining which action to take in order to resolve a divergence of graphs on multiple servers representing the same filesystem based on:

(4) a current master for the filesystem;

(5) a graph of snapnode objects for that filesystems global state;

(6) a list of current slave servers to that master for that filesystem.

16. A method according to claim 9 wherein the snapshot identifier identifies the time at which the snapshot was taken and the server on which the snapshot was taken.

17. A method or server according to claim 9 wherein a user interface is presented to a user for permitting access via a user to snapshots selected by a user.

18. A method according to claim 9 further comprising storing snapshots taken of the filesystem such that a previous snapshot of the filesystem can be selected by a user from the stored snapshots to restore the system to its state at the time of the selected snapshot.