Systems and methods for providing a quiescing protocol

• US 7,848,261 B2
• Filed: 02/17/2006
• Issued: 12/07/2010
• Est. Priority Date: 02/17/2006
• Status: Active Grant

• Current Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Original Assignee: Isilon Systems Incorporated (Dell Technologies Inc.)
• Inventors: Fachan, Neal T.
• Primary Examiner(s): Rao; Seema S
• Assistant Examiner(s): HUYNH, KHOA B
• Application Number:
  US11/357,740
• Publication Number:
  US 20070195810A1
• Time in Patent Office: 1,754 Days
• Field of Search: 370/241, 370/254, 370/229, 370/235, 370/236, 370/249, 370/252, 370/255, 709/223, 709/224, 709/225, 709/226, 709/220, 709/221, 709/222, 709/230, 709/232
• US Class Current: 370/255
• CPC Class Codes: H04L 41/30 Decision processes by autonomous network management units using voting and bidding
• CPC Class Codes: H04L 67/1001 for accessing one among a plurality of replicated servers
• CPC Class Codes: H04L 67/1029 using data related to the state of servers by a load balancer
• CPC Class Codes: H04L 67/1031 Controlling of the operation of servers by a load balancer, e.g. adding or removing servers that serve requests

Abstract

The systems and methods of the present invention provide a quiescing protocol. In one embodiment, nodes of a system utilize the protocol to complete processing until they reach a consistent state. In one embodiment, a coordinator initiates the quiescing process and the nodes communicate with each other to determine whether their messages have been processed and communicate with the coordinator to determine when all of the messages have been processed.

Claims

1. A method of quiescing a set of messages processed within a distributed system, the method comprising:

sending, by a computer processor, a first message to each of a plurality of nodes, the first message indicating that a quiescing protocol is to begin and that the nodes should prevent any new messages that change a state of the distributed system from being created;

receiving a second message from each of the plurality of nodes, each second message indicating that the node, which has sent that second message, has sent a third message to each of the plurality of nodes and that the node, which has sent that second message, has received a fourth message from each of the plurality of nodes, wherein the third message is a message requesting the fourth message, and wherein the fourth message indicates that the third message was received by the node to which it was sent and signifies that all messages that change a state of the distributed system previously sent by the node, which has sent that second message, to that node, which has sent that fourth message, have been processed;

determining, by the computer processor, that second messages have been received from all of the plurality of nodes;

based on the determination that second messages have been received from all of the plurality of nodes, sending, by the computer processor, a fifth message to each of the plurality of nodes, the fifth message requesting information about whether any new messages that change a state of the distributed system have been received from the plurality of nodes;

receiving a sixth message in response to the fifth message from each of the plurality of nodes, the sixth message including information related to the number of new messages, if any, that change a state of the distributed system that have been received, by the node which has sent the sixth message, from the plurality of nodes;

based on at least one received sixth message, determining, by the computer processor, that the distributed system has been quiesced if none of the plurality of nodes have received any new messages that change a state of the distributed system; and

if any of the plurality of nodes has received new messages that change a state of the distributed system, repeating the method of this claim until it is determined, by the computer processor, that none of the plurality of nodes have received any new messages that change a state of the distributed system.

2. The method of claim 1 performed by a coordinator node, the coordinator node comprising one of the plurality of nodes, wherein sending messages to each of the plurality of nodes includes sending local messages to the coordinator node and receiving messages from each of the plurality of nodes includes receiving local messages from the coordinator node.

3. The method of claim 1, further comprising:

if it is determined, by the computer processor, that none of the plurality of nodes have received new messages that change a state of the distributed system, sending a seventh message, by the computer processor, to each of the plurality of nodes, the seventh message indicating that the quiescing protocol is complete.

4. The method of claim 1, wherein sending messages to each of the plurality of nodes includes sending remote messages to at least one of the plurality of nodes and receiving messages from each of the plurality of nodes includes receiving remote messages from at least one of the plurality of nodes.

5. The method of claim 1, wherein the plurality of nodes manages a set of resources and the messages that change a state of the distributed system change a state of the distributed system by allocating the set of resources.

6. A method of quiescing a distributed system, the method comprising:

receiving, at a participant, a start message from a coordinator indicating that a quiescing protocol is to begin, the participant comprising a computer processor, and wherein the participant is one of a plurality of nodes in a distributed system;

suspending new messages that change a state of the distributed system sent by the participant;

setting, by the computer processor at the participant, a received message information to indicate that there are no messages that change a state of the distributed system received from the plurality of nodes;

receiving, at the participant, one or more messages that change a state of the distributed system from one or more of the plurality of nodes;

updating, by the computer processor at the participant, the received message information to indicate that at least one message that changes a state of the distributed system has been received from one or more of the plurality of nodes;

sending, by the computer processor from the participant, probe messages to the plurality of nodes, each probe message requesting a response;

receiving, at the participant, probe response messages from the plurality of nodes, each probe response message indicating that one of the probe messages was received from the participant and signifying that all messages sent by the participant to the responding node have been processed;

receiving, at the participant, probe messages from the plurality of nodes;

for each probe message received at the participant from the plurality of nodes, sending, by the computer processor from the participant, a corresponding probe response message from the participant to the sender of the probe message after the probe message is processed by the computer processor at the participant;

sending, by the computer processor from the participant, a checkpoint message to the coordinator indicating that the probe response messages have been received from the plurality of nodes;

receiving, at the participant, a continue message from the coordinator requesting the received message information;

sending, by the computer processor from the participant, the received message information to the coordinator; and

receiving, at the participant, a done message from the coordinator indicating that the quiescing protocol is complete.

7. The method of claim 6, wherein the coordinator is one of the plurality of nodes.

8. The method of claim 6, wherein sending probe messages to the plurality of nodes includes sending a local probe message to the participant.

9. The method of claim 6, wherein the plurality of nodes manages a set of resources and the at least one message that changes a state of the distributed system relates to the allocation of the set of resources.