System for live-migration and automated recovery of applications in a distributed system

• US 10,331,801 B2
• Filed: 12/17/2013
• Issued: 06/25/2019
• Est. Priority Date: 09/23/2011
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Open Invention Network LLC
• Inventors: Marsden, Luke
• Primary Examiner(s): Robinson, Greta L
• Application Number:
  US14/109,469
• Publication Number:
  US 20140108339A1
• Time in Patent Office: 2,016 Days
• Field of Search: 707615, 707616, 707623, 707633, 707634, 707635, 707637, 707638, 707639, 707674, 707686, 707E1701, 709208, 709209, 709214, 709223, 709211, 714 1, 714 12, 718105
• CPC Class Codes: G06F 11/0757 by exceeding a time limit, i.e. time-out, e.g. watchdogs
• CPC Class Codes: G06F 11/2028 eliminating a faulty processor or activating a spare
• CPC Class Codes: G06F 11/2038 with a single idle spare processing component
• CPC Class Codes: G06F 11/2094 Redundant storage or storage space G06F11/2056 takes precedence
• CPC Class Codes: G06F 11/2097 maintaining the standby controller/processing unit updated initialisation or re-synchronisation thereof G06F11/1658 and subgroups
• CPC Class Codes: G06F 11/3433 for load management allocation of a server based on load conditions G06F9/505; load rebalancing G06F9/5083; redistributing the load in a network by a load balancer H04L67/1029
• CPC Class Codes: G06F 16/214 Database migration support
• CPC Class Codes: G06F 16/273 Asynchronous replication or reconciliation
• CPC Class Codes: G06F 2201/84 Using snapshots, i.e. a logical point-in-time copy of the data

Abstract

A method and apparatus for distribution of applications amongst a number of servers, ensuring that changes to application data on a master for that application are asynchronously replicated to a number of slaves for that application. Servers may be located in geographically diverse locations; the invention permits data replication over high-latency and lossy network connections and failure-tolerance under hardware and network failure conditions. Access to applications is mediated by a distributed protocol handler which allows any request for any application to be addressed to any server, and which, when working in tandem with the replication system, pauses connections momentarily to allow seamless, consistent live-migration of applications and their state between servers. Additionally, a system which controls the aforementioned live-migration based on dynamic measurement of load generated by each application and the topological preferences of each application, in order to automatically keep servers at an optimum utilization level.

Claims

1. A system for dynamic migration of applications between servers, the system comprising:

a plurality of servers for hosting applications, the plurality of servers comprising at least a master server and one or more slave servers, one or more applications executing on said master server, each server comprising at least a protocol handler configured to receive requests for said one or more applications and configured to pause incoming requests for said one or more applications during a migration of said applications between servers, and the master server comprising master logic configured to replicate changes in application data hosted on the master server to one or more slave servers; and

a load balancer for measuring load on one of the plurality of servers caused by one or more applications hosted on that servers, the load balancer being configured to initiate migration of one or more applications from the master server to a slave server when a predetermined load condition of the master server is met;

wherein said migration of the one or more applications is comprised of mounting said one or more applications on said slave server using said application data replicated by said master logic to said slave server; and

wherein said migration is triggered when the load of the current master is greater than an average value of loads and a damping factor (Q).

2. A system or method according to claim 1 wherein the plurality of servers each has a controller that maintains a record of the server on which an application is currently hosted, and the protocol handler is configured to inspect the record to determine the server to which an incoming application request is to be directed.

3. A system or method according to claim 1 wherein the protocol handler is configured to pause incoming requests for an application and to terminate current requests for an application after a predetermined time period.

4. A system or method according to claim 1 wherein the protocol handler is configured to pause incoming requests for an application for a predetermined time period and to release the paused requests if current requests for an application have not completed in the predetermined time period.

5. Computer software which, when executed by appropriate processing means, causes the processing means to implement a method of managing a plurality of applications hosted by a cluster of servers which each have an interface connectable to at least one client by a network, each application delivering a service at the client, comprising:

electing a server of the cluster as a master server, the master server hosting at least one live application and electing a configurable number of servers in the cluster as slave servers;

replicating changes in application data of the live application hosted on the master server to said elected slave servers whereby each elected slave server maintains at least one version of the application data of the live application;

transferring hosting of the application from the master server to one of the elected slave servers in response to an event in the cluster, wherein the elected slave server mounts the application using its version of the application data and becomes a new master server;

wherein said event in the cluster comprises detecting an alternative master server in the cluster based on the load of the current master being greater than an average value of loads of all of the servers in the cluster and a damping factor (Q).

6. Computer software according to claim 5 wherein the event is detection of a preferred alternate master server based on the locality of servers in the cluster.

7. Computer software according to claim 5 wherein the event is detection of a preferred alternate master server in the cluster based on a predefined user preference.

8. Computer software according to any of claim 5 wherein the event is detected by exchanging messages with other servers of the cluster.

9. Computer software according to any of claim 5 wherein the event is addition of a server to the cluster.

10. Computer software according to claim 9 wherein the cluster included a single server prior to addition of the one server.

11. Computer software according to claim 9 wherein on addition of a server to the cluster, the new load of the servers is determined, and a decision as to which of the applications hosted by the cluster should be migrated to the newly added server is made.

12. Computer software according of any of claim 5 wherein the event is removal of a server from the cluster, wherein the removal was anticipated.

13. Computer software according to any of claim 5 wherein the event is failure of a server in the cluster, wherein the data for live applications hosted by the failed server is recovered using versions of the current application on servers in the cluster which are continuing to operate.

14. Computer software according to any of claim 5 wherein the event is a partition of the cluster, and wherein after recovery from the partition a preferred alternate master server is selected from a number of potentially competing master servers as the server with the version of the current application data which is more valuable.

15. Computer software according to any of claim 5 wherein a leader function hosted on one of the servers determines the new master server for the application, wherein the leader function is on a different server from the master.

16. Computer software according to any of claim 5 wherein a leader function hosted on the current master server determines the new master server.

17. Computer software which, when executed by appropriate processing means, causes the processing means to implement a system for dynamic migration of applications between servers, the system comprising:

one or more servers comprising a master server and one or more slave servers, one or more applications on each of said master server and one or more slave servers, a protocol handler configured to receive requests for said one or more applications and configured to pause incoming requests for said one or more applications during a migration of applications between servers, and master logic configured to replicate changes in application data hosted on the master server to one or more of said slave servers;

a load balancer configured to measure load on the one or more servers caused by the one or more applications, and configured to initiate a migration of one or more applications from the master server to a slave server when the load of the master server is greater than an average value of loads and a damping factor (Q); and

wherein said migration is comprised of mounting said one or more applications on said slave server using said application data replicated by said master logic.

18. A system, comprising:

one or more servers comprising a master server and one or more slave servers, one or more applications on each of said master server and one or more slave servers, a protocol handler configured to receive requests for said one or more applications and configured to pause incoming requests for said one or more applications during a migration of applications between servers, and master logic configured to replicate changes in application data hosted on the master server to one or more of said slave servers;

a load balancer configured to measure load on the one or more servers caused by the one or more applications, and configured to initiate a migration of one or more applications from the master server to a slave server when the load of the master server is greater than an average value of loads and a damping factor (Q); and

wherein said migration is comprised of mounting said one or more applications on said slave server using said application data replicated by said master logic.