I/O scheduling and load balancing across the multiple nodes of a clustered environment

• US 9,170,928 B1
• Filed: 12/31/2013
• Issued: 10/27/2015
• Est. Priority Date: 12/31/2013
• Status: Active Grant

• Current Assignee: Veritas Technologies, LLC (Whitehouse Group Ltd.)
• Original Assignee: Symantec Corporation (NortonLifeLock Inc.)
• Inventors: Dantkale, Suhas, Thakur, Satyendra, Kaliannan, Kirubakaran, Vadlamannati, Prasad
• Primary Examiner(s): Mamo, Elias
• Application Number:
  US14/145,875
• Time in Patent Office: 665 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 11/2066 Optimisation of the communication load
• CPC Class Codes: G06F 12/0223 User address space allocation, e.g. contiguous or non contiguous base addressing
• CPC Class Codes: G06F 2201/855 Details of asynchronous mirroring using a journal to transfer not-yet-mirrored changes
• CPC Class Codes: G06F 3/061 Improving I/O performance
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/065 Replication mechanisms
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: H04L 67/1008 based on parameters of servers, e.g. available memory or workload monitoring of computer activity G06F11/30
• CPC Class Codes: H04L 67/1014 based on the content of a request
• CPC Class Codes: H04L 67/1095 Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

Write operations are scheduled for multiple nodes in a shared storage cluster that supports volume replication. Requests are received from nodes for allocation of space for write operations in a replication log. In response to a received request, the current capacity of the requesting node to manage a backlog can be determined. The amount of space in the replication log allocated to the node is then calibrated to the node'"'"'s capacity, thereby preventing self-throttling. A separate priority can be assigned to each volume, and space in the replication log assigned to each volume based on its priority. Nodes can target synchronous and other latency sensitive operations to higher priority volumes. A single global queue can be maintained to schedule write operations for all nodes, thereby providing a fair scheduling. A separate local queue can be maintained for each node, thereby providing specific levels of preference to specific nodes.

Claims

1. A computer implemented method for scheduling write operations of a plurality of nodes, thereby balancing a load of storage input/output operations in a clustering and storage environment that utilizes shared storage and supports replication of data volumes, the method comprising the steps of:

receiving, by a computer, requests from nodes of the clustering and storage environment for allocation of space for write operations in a replication log, wherein nodes perform write operations by writing first to the replication log and subsequently to the shared storage, wherein an order of write operations to the replication log matches an order of write operations to the shared storage, and wherein the clustering and storage environment utilizes the order and content of write operations to the replication log to replicate the data volumes of the shared storage to a secondary site, thereby preserving write-order fidelity;

responsive to receiving a request from a specific node for the allocation of space in the replication log, determining, by the computer, a current capacity of the specific node to manage a backlog of write operations without throttling execution of its own write operations; and

calibrating, by the computer, an amount of space in the replication log to allocate to the determined current capacity of the specific node.

2. The method of claim 1 wherein calibrating an amount of space in the replication log to allocate to the determined current capacity of the specific node further comprises:

allocating the specific node only enough space in the replication log to execute a number of backlogged write operations that the specific node can simultaneously manage without throttling its own write operations.

3. The method of claim 1 wherein calibrating an amount of space in the replication log to allocate to the determined current capacity of the specific node further comprises:

limiting allocation of space in the replication log to an upper limit of the specific node for simultaneous backlogged write operation management, thereby preventing the specific node from throttling its backlogged write operations above the upper limit.

4. The method of claim 1 wherein determining a current capacity of a specific node to manage a backlog of write operations without throttling execution of its own write operations further comprises:

gleaning information indicative of the current capacity of the specific node from a request received from the specific node for allocation of space for write operations in the replication log, wherein each received request for allocation of space in the replication log further comprises information indicative of the current capacity of the requesting node.

5. The method of claim 4 wherein the information indicative of the current capacity of the requesting node in received requests further comprises:

a number of outstanding storage input/out requests pending on the node, and the node'"'"'s Small Computer System Interface (“

SCSI”

) tag command queue.

6. The method of claim 1 wherein receiving requests from nodes of the clustering and storage environment for allocation of space for write operations further comprises:

receiving requests from nodes of the clustering and storage environment for allocation of space for write operations, each one of said received requests further comprising information indicative of a priority of the corresponding write operation; and

scheduling write operations at a level of priority indicated in corresponding received requests.

7. The method of claim 1 wherein determining a current capacity of a specific node to manage a backlog of write operations without throttling execution of its own write operations further comprises:

periodically polling each one of the plurality of nodes for information indicative of the current capacity of the specific node.