Automatic clusterwide fail-back

• US 8,443,232 B1
• Filed: 03/31/2011
• Issued: 05/14/2013
• Est. Priority Date: 10/28/2005
• Status: Active Grant

• Current Assignee: Veritas Technologies, LLC (Whitehouse Group Ltd.)
• Original Assignee: Symantec Operating Corporation (Gen Digital Inc.)
• Inventors: Nagineni, Venkata Sreenivasa Rao, Nandi, Siddhartha, Singh, Abhay K.
• Primary Examiner(s): Riad, Amine
• Application Number:
  US13/076,974
• Time in Patent Office: 775 Days
• Field of Search: 714/13
• US Class Current: 714/13
• CPC Class Codes: G06F 11/2033 switching over of hardware resources
• CPC Class Codes: G06F 11/2035 without idle spare hardware
• CPC Class Codes: G06F 11/2046 where the redundant components share persistent storage G06F11/2043 takes precedence
• CPC Class Codes: G06F 11/2092 Techniques of failing over between control units

Abstract

Systems and procedures may be used to coordinate the fail-back of multiple hosts in environments where the hosts share one or more data-storage resources. In one implementation, a procedure for coordinating fail-backs includes monitoring a failed data path to detect a restoration of the data path, polling remaining nodes in response to the restoration, and allowing the first node to resume communications if access has been restored to the remaining nodes.

Claims

1. A method comprising:

determining that access by a first node in a computation cluster through a first controller to a storage resource has been restored;

querying at least one of a plurality of active nodes in the computation cluster, wherein

the querying is performed in response to the determining that access by the first node through the first controller to the storage resource has been restored, and

the active nodes in the computation cluster are nodes in the computation cluster that depend on access to the storage resource at the time of the querying, and for which the first controller is a fail-back target;

receiving confirmation, in response to the querying, that access to the storage resource through the first controller has been restored for each of the active nodes in the computation cluster; and

instructing, based at least in part on the confirmation, the first node to communicate with the storage resource through the first controller.

2. The method of claim 1, comprising:

querying a second node in the computation cluster to determine whether the first controller is a fail-back target for the second node; and

instructing, based at least in part on the confirmation, at least one of the active nodes in the computation cluster to communicate with the storage resource through the first controller.

3. The method of claim 1, wherein:

storage resource comprises a logical unit (LUN) or a group of logical units on a storage array;

the confirmation is generated by a master node in the computation cluster;

the querying is performed by the master node; and

the instructing is performed by the master node.

4. The method of claim 3, wherein the master node comprises the first node.

5. The method of claim 3, wherein the master node comprises one of the active nodes in the computation cluster.

6. The method of claim 1, comprising:

monitoring a first data path from the first node through the first controller to the storage resource, wherein

the determining that access by the first node through the first controller to the storage resource has been restored is performed based at least in part on the monitoring.

7. The method of claim 6, comprising:

detecting a failure of the first data path; and

communicating between the first node and the storage resource through a second controller prior to the determining that access by the first node through the first controller to the storage resource has been restored, wherein

the monitoring comprises transmitting detection commands from the first node through a port on the first node, wherein the port is coupled to the first data path.

8. The method of claim 7, wherein the detection commands comprise at least one of:

a periodic heartbeat signal, or an inquiry command to a storage array that comprises the controller, or a read command to the storage array, or a write command to the storage array.

9. The method of claim 1, wherein the determining that access by the first node through the first controller to the storage resource has been restored comprises:

receiving, a master node in the cluster, a fail-back request from the first node.

10. The method of claim 1, wherein the determining that access by the first node through the first controller to the storage resource has been restored comprises:

informing a master node in the cluster that the access has been restored, wherein the informing is performed by the first node.

11. A method comprising:

determining that a first data path is available for use by a first node in a computation cluster, wherein

the computation cluster comprises a plurality of active nodes,

the active nodes are nodes that depend on access to a destination, and for which the first data path is a fail-back target; and

the first data path was unavailable to the first node prior to the determining;

informing a master node in the computation cluster that the first data path is available for use by the first node;

abstaining from communicating between the first node and the destination over the first data path until a fail-back approval is received from the master node, wherein

the abstaining occurs after the determining that the first data path is available for use by the first node;

the fail-back approval is received after the informing, and

the fail-back approval is based at least in part on a determination that the first data path is available for use by each of the plurality of active nodes; and

communicating between the first node and the destination after the fail-back approval is received from the master node.

12. The method of claim 11, wherein:

the determining is performed by the first node; and

the destination comprises a storage resource.

13. The method of claim 11, wherein:

the destination comprises a logical unit (LUN) or a group of logical units on a storage array.

14. The method of claim 11, wherein the determining comprises:

transmitting detection commands from the first node through a port on the first node coupled to the first data path; and

receiving, at the first node, a response to the detection commands.

15. A system comprising:

a first host comprising

a restoration detection module configured to monitor a failed communications path to a storage resource, and

a query module configured to transmit a request for approval for a fail-back, wherein

a plurality of hosts are coupled to the failed communications path, and

the query module is configured to transmit the request in response to the restoration detection module detecting a restoration of the failed communications path; and

a master host coupled to the first host, the master host comprising

a coordination module configured to

receive the request from the first host,

determine, in response to the request, whether each active host in the plurality of hosts is ready to perform a fail-back, wherein

the active hosts are hosts that depend on access to the storage resource, and

transmit a fail-back approval to the first host, wherein

the fail-back approval is based at least in part the request, and

the first host is configured to abstain from communicating over the failed communications path until receiving the fail-back approval.

16. The system of claim 15, wherein the plurality of hosts comprises the master host, and the storage resource comprises a logical unit (LUN) or a group of logical units on a storage array.

17. The system of claim 15, wherein the coordination module is configured to transmit a fail-back approval to the first host only if each active host in the plurality of hosts is ready to perform a fail-back.

18. The system of claim 15, wherein:

the restoration detection module is configured to transmit detection commands through a port on the first host, wherein

the port is coupled to the failed communications path, and

the detection commands comprise at least one of;

a periodic heartbeat signal, or an inquiry command to a storage array that comprises the storage resource, or a read command to the storage array, or a write command to the storage array; and

the restoration detection module is configured to detect a restoration of the failed communications path based upon a response to the detection commands.

19. A non-transient computer readable medium having encoded therein instructions executable on one or more processors, wherein the instructions are configured to implement each of:

determining that access by a first node in a computation cluster through a first controller to a storage resource has been restored;

querying at least one of a plurality of active nodes in the computation cluster, wherein

the querying is performed in response to the determining that access by the first node through the first controller to the storage resource has been restored, and

the active nodes in the computation cluster are nodes in the computation cluster that depend on access to the storage resource at the time of the querying, and for which the first controller is a fail-back target; and

receiving confirmation, in response to the querying, that access to the storage resource through the first controller has been restored for each of the active nodes in the computation cluster.

20. The computer readable medium of claim 19, wherein the instructions are configured to implement each of:

instructing, based at least in part on the confirmation, the first node to communicate with the storage resource through the first controller.