Distributed workload reassignment following communication failure

• US 9,847,918 B2
• Filed: 08/12/2014
• Issued: 12/19/2017
• Est. Priority Date: 08/12/2014
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Kruse, David, Zhu, Qibo, Dion, David, Petter, Vladimir, Koppolu, Lokesh Srinivas
• Primary Examiner(s): NGUYEN, THU HA T
• Application Number:
  US14/457,842
• Publication Number:
  US 20160050294A1
• Time in Patent Office: 1,225 Days
• Field of Search: 709201, 709208
• CPC Class Codes: G06F 11/2046 where the redundant components share persistent storage G06F11/2043 takes precedence
• CPC Class Codes: G06F 16/1774 Locking methods, e.g. locking methods for file systems allowing shared and concurrent access to files
• CPC Class Codes: G06F 16/182 Distributed file systems
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: G06F 9/5027 the resource being a machine, e.g. CPUs, Servers, Terminals
• CPC Class Codes: G06F 9/52 Program synchronisation; Mutual exclusion, e.g. by means of semaphores
• CPC Class Codes: H04L 43/00 Arrangements for monitoring or testing data switching networks
• CPC Class Codes: H04L 67/1001 for accessing one among a plurality of replicated servers
• CPC Class Codes: H04L 67/1029 using data related to the state of servers by a load balancer
• CPC Class Codes: H04L 67/1034 Reaction to server failures by a load balancer
• CPC Class Codes: H04L 67/61 taking into account QoS or priority requirements
• CPC Class Codes: H04L 69/40 for recovering from a failure of a protocol instance or entity, e.g. service redundancy protocols, protocol state redundancy or protocol service redirection management of faults, events, alarms or notifications in data switching networks H04L41/06

Abstract

A generation identifier is employed with various systems and methods in order to identify situations where a workload has been reassigned to a new node and where a workload is still being processed by an old node during a failure between nodes. A master node may assign a workload to a worker node. The worker node sends a request to access target data. The request may be associated with a generation identifier and workload identifier that identifies the node and workload. At some point, a failure occurs between the master node and worker node. The master node reassigns the workload to another worker node. The new worker node accesses the target data with a different generation identifier, indicating to the storage system that the workload has been reassigned. The old worker node receives an indication from the storage system that the workload has been reassigned and stops processing the workload.

Claims

1. A system, comprising:

at least one processor;

memory, operatively connected to the at least one processor and containing instructions that, when executed by the at least one processor, perform a method, the method comprising;

associating, by a master node, a first generation identifier and a first workload identifier with a first workload, wherein the first generation identifier communicates an initial priority associated with a first worker node;

communicating the first generation identifier, the first workload identifier, and the first workload to the first worker node; and

during a communication failure with the first worker node, generating a second generation identifier, associating the second generation identifier with the first workload, and communicating the second generation identifier, the first workload identifier, and the first workload to a second worker node, wherein the second generation identifier communicates a different priority from the first generation identifier.

2. The system of claim 1, wherein the communication failure comprises at least one of:

a disconnection, a hardware failure, or a reboot of a node.

3. The system of claim 1, wherein the second generation identifier communicates a higher priority than the first generation identifier.

4. The system of claim 1, wherein the second generation identifier communicates a lower priority than the first generation identifier.

5. The system of claim 4, further comprising:

receiving an indication of success from the second worker node;

generating a third generation identifier communicating a higher priority than the first generation identifier;

associating the third generation identifier with the first workload; and

communicating the third generation identifier to the second worker node.

6. The system of claim 1, wherein communicating the second generation identifier comprises sending an indication that the second worker node should not interrupt the first worker node if it is determined that the first worker node is still processing the first workload.

7. The system of claim 1, wherein the first and second generation identifiers are each comprised of at least two subparts, each of the at least two subparts communicating a priority.

8. The system of claim 7, wherein the priority communicated by each of the at least two subparts is weighted such that an indication of priority by one of the at least two subparts preempts priority indicated by another subpart of the at least two subparts.

9. A computer implemented method, the method comprising:

receiving a first access request for target data, the target data stored by a storage system, and the first access request associated with a first generation identifier and a first workload identifier;

storing the first generation identifier and the first workload identifier in persistent storage of the storage system as a first stored generation identifier and a first stored workload identifier;

granting the first access request for the target data;

receiving a second access request for the target data, the second access request associated with a second generation identifier and the first workload identifier;

comparing the second generation identifier to the first stored generation identifier and the first workload identifier to the first stored workload identifier;

determining that the second generation identifier indicates a higher priority than the first stored generation identifier, and that the first workload identifier matches the first stored workload identifier;

storing the second generation identifier in persistent storage as a second stored generation identifier;

granting the second access request for the target data;

receiving a third access request for the target data, the third access request associated with the first generation identifier and the first workload identifier;

determining that the first generation identifier indicates a lower priority than the second stored generation identifier and that the first workload identifier matches the first stored workload identifier; and

denying the third access request for the target data.

10. The method of claim 9, wherein the first access request and the third access request are received from a first node and the second access request is received from a second node, and denying the third access request for the target data comprises providing an indication to the first node that the first workload has been reassigned to the second node.

11. The method of claim 9, wherein denying the third access request comprises providing an indication that the first generation identifier indicates a lower priority than the second stored generation identifier.

12. The method of claim 9, wherein the third access request is received from a first node, and wherein the method further comprises:

receiving a fourth access request for the target data from a second node, the fourth access request associated with a third generation identifier and the first workload identifier, and comprising an indication that the request is permissive;

determining that the third generation identifier indicates a higher priority than the second stored generation identifier;

determining that the storage system can still communicate with the first node; and

denying the fourth access request for the target data.

13. The method of claim 12, wherein denying the fourth access request comprises providing an indication to the second node that the storage system can still communicate with the first node.

14. The method of claim 9, further comprising:

locking the target data upon granting the first access request; and

after granting the second access request, releasing the lock on the target data and removing the first stored generation identifier from persistent storage.

15. The method of claim 9, wherein the first, second, and third generation identifiers are each comprised of at least two subparts, each of the at least two subparts communicating a priority.

16. The method of claim 9, wherein the priority communicated by each of the at least two subparts is weighted such that an indication of priority by one of the at least two subparts preempts priority indicated by another subpart of the at least two subparts.

17. A computer implemented method, the method comprising:

sending a first access request for target data to a storage system, the first access request associated with a first generation identifier and a first workload identifier associated with a first workload executing on a first node;

receiving a first access response granting access to the target data;

after receiving the first access response, sending an indication to a second node that access to the target data was granted;

receiving a second generation identifier for use in subsequent access requests;

sending a second access request for target data to the storage system, the second access request associated with the second generation identifier and the first workload identifier; and

receiving a second access response granting access to the target data.

18. The method of claim 17, further comprising:

sending a third access request for target data to the storage system, the third access request associated with the second generation identifier and the first workload identifier;

receiving a third access response denying access to the target data and comprising an indication that access to the target data for the first workload has been reassigned to a third node; and

stopping execution of the first workload on the first node as a result of the received third access response.

19. The method of claim 18, wherein the third access response is received after a communication failure between the first node and the second node.

20. The method of claim 17, wherein the first access request comprises an indication that the first access request is permissive.