SCALABLE FAULT RESILIENT COMMUNICATIONS WITHIN DISTRIBUTED CLUSTERS

• US 20160050262A1
• Filed: 08/13/2014
• Published: 02/18/2016
• Est. Priority Date: 08/13/2014
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Kumar, Abhinit, D'Amato, Andrea, Dion, David Allen, Nishanov, Gor, Koppolu, Lokesh Srinivas, Maliwacki, Nicholas
• Granted Patent: US 11,290,524 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 9/4856 resumption being on a different machine, e.g. task migration, virtual machine migration G06F9/5088 takes precedence
• CPC Class Codes: G06F 9/5083 Techniques for rebalancing the load in a distributed system
• CPC Class Codes: G06F 9/5088 involving task migration
• CPC Class Codes: H04L 41/0654 using network fault recovery ring fault isolation or reconfiguration in loop networks without recovery actions by a network management system H04L12/437
• CPC Class Codes: H04L 43/0817 by checking functioning
• CPC Class Codes: H04L 43/0823 Errors, e.g. transmission errors
• CPC Class Codes: H04L 43/0876 Network utilisation, e.g. volume of load or congestion level
• CPC Class Codes: H04L 43/10 Active monitoring, e.g. heartbeat, ping or trace-route
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46

Abstract

Embodiments are directed to communicating between computing nodes in a cluster of nodes. In one scenario, a computer system receives a data packet from a worker node including the worker node'"'"'s current workload identifiers and health status, where the data packet includes an associated version number. The computer system determines that the version number in the received data packet is different than a previously received data packet and evaluates the worker node'"'"'s current workload configuration to determine whether workload changes are to be made on the worker node. Then, upon determining that workload changes are to be made on the worker node, the computer system selects a subset of workload changes to apply to the worker node, generates an indication of the selected subset of workload changes to the worker node and sends the generated indication of workload changes to the worker node.

Claims

1. At a computer system including at least one processor, a computer-implemented method for communicating between computing nodes in a cluster of nodes, the method comprising:

an act of receiving a data packet from a worker node including the worker node'"'"'s current workload identifiers and health status, the data packet including an associated version identifier;

an act of determining that the version identifier in the received data packet is different than a previously received data packet;

an act of evaluating the worker node'"'"'s current workload configuration to determine whether workload changes are to be made on the worker node;

upon determining that workload changes are to be made on the worker node, an act of selecting a subset of workload changes to apply to the worker node;

an act of generating an indication of the selected subset of workload changes to the worker node; and

an act of sending the generated indication of workload changes to the worker node.

2. The method of claim 1, wherein the generated indication of workload changes are sent to the worker node via a fault tolerant communication protocol.

3. The method of claim 2, wherein the fault tolerant communication protocol specifies an upper bound on the number of data packets that are transmitted or received by the computing system per time period.

4. The method of claim 1, wherein the workload changes comprise at least one of adding workloads to the worker node, removing workloads from the worker node or changing one or more workloads on the worker node.

5. The method of claim 1, further comprising:

an act of determining that a bandwidth limitation applies to the indication of workload changes; and

upon determining that a bandwidth limitation applies to the indication of workload changes, an act of dividing the selected subset of workload changes into at least two indication messages.

6. The method of claim 1, wherein evaluating the worker node'"'"'s current workload configuration to determine whether workload changes are to be made on the worker node comprises comparing the worker node'"'"'s current workload configuration against a master node'"'"'s indication of which workloads are to be processed by the worker node.

7. The method of claim 6, wherein the generated indication of workload changes to the worker node comprises the delta between the worker node'"'"'s indication of workload configuration and the master node'"'"'s indication of which workloads are to be processed by the worker node.

8. The method of claim 1, wherein a bandwidth limit is specified for each protocol that is used to communicate with the computing system or with the worker node.

9. The method of claim 1, wherein the generated indication of workload changes is sent to the worker node as part of a status query message.

10. The method of claim 9, wherein the status query message is sent to the worker node a specified interval, the specified interval corresponding to a priority level for the workload being processed by the worker node.

11. The method of claim 10, wherein the specified interval at which the status query message is sent to the worker node changes dynamically according to the priority level specified for the workload.

12. At a computer system including at least one processor, a computer-implemented method for communicating between computing nodes in a cluster of nodes, the method comprising:

an act of generating a data packet that includes a listing of workload identifiers (IDs) for which a worker node is responsible, the data packet further including a version identifier and workload health status for each workload ID;

an act of sending the generated data packet to a master node in the cluster of nodes;

an act of receiving a delta data packet that includes information specifying a subset of selected changes that are to be applied to at least one of the worker node'"'"'s workloads; and

an act of applying the selected subset of workload changes specified in the delta data packet to the at least one workload on the worker node.

13. The method of claim 12, wherein the generated data packet is sent to the master node at a randomized time, such that data packets from multiple worker nodes are sent at different times.

14. The method of claim 13, wherein the randomized time is based on a specified heartbeat time interval with an added delta time that is between zero and the specified heartbeat time interval.

15. The method of claim 12, wherein the generated data packet is sent using a fault tolerant communication protocol that has been limited by the master node to a specified amount of bandwidth per time period.

16. The method of claim 12, further comprising:

an act of sending a status query packet to the master node to verify that a physical link between the computing system and the master node is functioning properly;

an act of determining that a specified period of time has passed without receiving an acknowledgement packet from the master node; and

an act of decreasing a reliability score for the master node, the reliability score indicating the likelihood that the physical link between the master node and the computing system is functioning properly.

17. The method of claim 16, further comprising:

an act of sending a second status query packet to the master node to verify that the physical link between the computing system and the master node is functioning properly;

an act of determining that an acknowledgement packet has been received from the master node; and

an act of increasing the reliability score for the master node to a maximum level, indicating that the physical link between the master node and the computing system is functioning properly.

18. The method of claim 16, further comprising:

an act of determining that a threshold number of master nodes have failed or become unavailable; and

an act of dynamically decreasing the amount of time that is allowed to pass without receiving an acknowledgement packet from the master node before the reliability score is reduced.

19. A computer system comprising the following:

one or more processors;

one or more computer-readable storage media having stored thereon computer-executable instructions that, when executed by the one or more processors, cause the computing system to perform a method for communicating between computing nodes in a cluster of nodes, the method comprising;

an act of receiving a data packet from a worker node including the worker node'"'"'s current workload identifiers and health status, the data packet including an associated version number;

an act of determining that the version number in the received data packet is different than a previously received data packet;

an act of evaluating the worker node'"'"'s current workload configuration to determine whether workload changes are to be made on the worker node;

upon determining that workload changes are to be made on the worker node, an act of selecting a subset of workload changes to apply to the worker node;

an act of generating an indication of the selected subset of workload changes to the worker node; and

an act of sending the generated indication of workload changes to the worker node.

20. The computer system of claim 19, wherein the generated indication of workload changes are sent to the worker node via a fault tolerant communication protocol, and wherein the fault tolerant communication protocol specifies an upper bound on the number of data packets that are transmitted or received by the computing system per time period.