Performing I/O quiesce and drain operations in multi-node distributed systems

• US 9,645,859 B1
• Filed: 12/18/2012
• Issued: 05/09/2017
• Est. Priority Date: 12/18/2012
• Status: Active Grant

• Current Assignee: Veritas Technologies, LLC (Whitehouse Group Ltd.)
• Original Assignee: Veritas Technologies, LLC (Whitehouse Group Ltd.)
• Inventors: Dash, Prasanta R., Randhawa, Amarinder Singh, Jagtap, Asmita, Yalamanchili, Chaitanya, Buddhi, Madhav
• Primary Examiner(s): Dang, Khanh
• Application Number:
  US13/718,740
• Time in Patent Office: 1,603 Days
• Field of Search: 710 28, 710 32, 710 36, 710 39, 710 40, 710310, 710240, 710244, 711147, 711148, 711150-152, 711162
• CPC Class Codes: G06F 3/00 Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
• CPC Class Codes: G06F 9/52 Program synchronisation; Mutual exclusion, e.g. by means of semaphores

Abstract

Various systems and methods for performing Input/Output (I/O) quiesce and drain operations in multi-node distributed storage systems are disclosed. For example, one method involves receiving a message. The message indicates a request for a operation to be performed by a node, where the node can receive I/O requests from an application, as well as remote I/O requests from another node. The node can issue the I/O requests and the remote I/O requests to the one or more storage devices. In response to receipt of the message, the method performs a first portion of a operation on the node. The first portion of the operation includes the node not processing any additional locally generated I/O requests and processing additional remote I/O requests.

Claims

1. A method comprising:

receiving a message, wherein

the message indicates a request for an operation to be performed by a first node,

the operation comprises a quiesce portion and a drain portion,

a plurality of nodes comprises the first node and a second node,

the first node is configured to receive one or more first input/output (I/O) requests from a first application,

the first node is configured to receive one or more second I/O requests from the second node, and

the first node is configured to issue the one or more first I/O requests and the one or more second I/O requests to one or more storage devices;

in response to receipt of the message, performing the quiesce portion of the operation on the first node, wherein

the quiesce portion of the operation comprises preventing the first node from processing additional first I/O requests from the first application, and

the drain portion of the operation comprises causing the first node to process additional second I/O requests from the second node;

determining whether the quiesce portion of the operation is completed; and

in response to determining that the quiesce portion of the operation is completed,

performing the drain portion of the operation, and

sending an indication of completion of the drain portion of the operation to a control node, wherein

the control node is configured to receive indications of completion from each of the plurality of nodes.

2. The method of claim 1, wherein

the control node is further configured to determine whether indications of completions are received from all of the plurality of nodes, and

the control node is further configured to, in response to a determination that indications of completions are received from all of the plurality of nodes, indicate a completion of the quiesce portion of a system-wide operation.

3. The method of claim 1, wherein

the drain portion of the operation comprises the first node communicating the first I/O requests and the second I/O requests to the one or more storage devices, and

the drain portion of the operation further comprises the first node receiving additional second I/O requests.

4. The method of claim 3, wherein

the control node is further configured to determine whether indications of completions are received from all of the plurality of nodes, and

the control node is further configured to, in response to a determination that indications of completions are received from all of the plurality of nodes, indicate a completion of the drain portion of a system-wide operation.

5. The method of claim 3, wherein

the one or more first I/O requests are generated by the first application,

the one or more first I/O requests are associated with a first data object,

the performing the drain portion of the operation further comprises

determining whether to process a first additional second I/O request of the additional second I/O requests, wherein

the determining comprises determining whether the first additional second I/O request is associated with the first data object;

and

in response to a determination that the first additional I/O request is associated with the first data object, processing the first additional second I/O request.

6. The method of claim 4, wherein

the control node is further configured to send an indication of the completion of the drain portion of the system-wide operation to each of the plurality of nodes;

and

each of the plurality of nodes are configured to resume normal operations subsequent to receiving the indication.

7. The method of claim 1, wherein

the second node is configured to receive the one or more second I/O requests from a second application.

8. The method of claim 1, wherein

the first node not processing the additional first I/O requests comprises

the first node receiving the additional first I/O requests,

the first node storing the additional first I/O requests, and

the first node not communicating the additional first I/O requests to the one or more storage devices.

9. The method of claim 1, wherein

the first node processing the additional second I/O requests comprises

the first node receiving the additional second I/O requests,

the first node communicating the additional second I/O requests to the one or more storage devices.

10. The method of claim 1, wherein

the first node processing the additional second I/O requests comprises

the first node not receiving the additional second I/O requests,

the first node communicating an error message to the second node, wherein

the error message indicates to the second node that the first node is not processing the additional second I/O requests.

11. The method of claim 10, wherein

the second node is configured to

in response to receiving the error message, communicating the additional second I/O requests to at least a third node of the plurality of nodes.

12. The method of claim 1, further comprising:

configuring the first node to not process the additional second I/O requests.

13. The method of claim 1,

determining a third node of the plurality of nodes, wherein

the third node of the plurality of nodes is configured to process second I/O requests in response to receiving the message.

14. A system comprising:

one or more processors,

a control module of a first node, wherein

the control module of the first node is configured to

receive a message, wherein

the message indicates a request for an operation to be performed by the first node,

the operation comprises a quiesce portion and a drain portion,

a plurality of nodes comprises the first node and a second node,

the first node is configured to receive one or more first input/output (I/O) requests from a first application,

the first node is configured to receive one or more second I/O requests from the second node, and

the first node is configured to issue the one or more first I/O requests and the second I/O requests to one or more storage devices,

in response to receipt of the message, perform the quiesce portion of the operation on the first node, wherein

the quiesce portion of the operation comprises preventing the first node from processing additional first I/O requests from the first application, and

the drain portion of the operation comprises causing the first node to process additional second I/O requests from the second node;

determine whether the quiesce portion of the operation is completed; and

in response to determining that the quiesce portion of the operation is completed,

perform the drain portion of the operation, and

send an indication of completion of the drain portion of the operation to a control node, wherein

 

the control node is configured to receive indications of completion from each of the plurality of nodes; and

the control module of the first node is configured to be executed using the one or more processors.

15. The system of claim 14, wherein

the control node is further configured to determine whether indications of completions are received from all of the plurality of nodes, and

the control node is further configured to, in response to a determination that indications of completions are received from all of the plurality of nodes, indicate a completion of the quiesce portion of a system-wide operation.

16. The system of claim 14, wherein

the drain portion of the operation comprises the first node communicating the first I/O requests and the second I/O requests to the one or more storage devices,

the drain portion of the operation further comprises the first node receiving additional second I/O requests.

17. The system of claim 14, wherein

the first node not processing the additional first I/O requests comprises

the first node receiving the additional first I/O requests,

the first node storing the additional first I/O requests, and

the first node not communicating the additional first I/O requests to the one or more storage devices.

18. A computer program product comprising:

a plurality of instructions, comprising

a first set of instructions, executable on a computer system, configured to receive a message, wherein

the message indicates a request for an operation to be performed by a first node,

the operation comprises a quiesce portion and a drain portion;

a plurality of nodes comprises the first node and a second node,

the first node is configured to receive one or more first input/output (I/O) requests from a first application,

the first node is configured to receive one or more second I/O requests from the second node, and

the first node is configured to issue the one or more first I/O requests and the second I/O requests to one or more storage devices,

a second set of instructions, executable on the computer system, configured to perform the quiesce portion of the operation on the first node, in response to receipt of the message, wherein

the quiesce portion of the operation comprises preventing the first node from processing additional first I/O requests from the first application, and

the drain portion of the operation further comprises causing the first node to process additional second I/O requests from the second node,

a third set of instructions, executable on the computer system, configured to determine whether the quiesce portion of the operation is completed, and

a fourth set of instructions, executable of the computer system, configured to perform the drain portion of the operation in response to determining that the quiesce portion of the operation is completed, and

send an indication of completion of the drain portion of the operation to a control node, wherein

the control node is configured to receive indications of completion from each of the plurality of nodes; and

a non-transitory computer-readable storage medium, wherein the instructions are encoded in the non-transitory computer-readable storage medium.

19. The computer program product of claim 18, wherein

the control node is further configured to determine whether indications of completions are received from all of the plurality of nodes, and

the control node is further configured to, in response to a determination that indications of completions are received from all of the plurality of nodes, indicate a completion of the quiesce portion of a system-wide operation.

20. The computer program product of claim 18, wherein

the drain portion of the operation comprises the first node communicating the first I/O requests and the second I/O requests to the one or more storage devices, and

the drain portion of the operation further comprises the first node receiving additional second I/O requests.

21. The computer program product of claim 18, wherein

the first node not processing the additional first I/O requests comprises

the first node receiving the additional first I/O requests,

the first node storing the additional first I/O requests, and

the first node not communicating the additional first I/O requests to the one or more storage devices.