System and method for resolving cluster partitions in out-of-band storage virtualization environments

• US 7,739,541 B1
• Filed: 07/25/2003
• Issued: 06/15/2010
• Est. Priority Date: 07/25/2003
• Status: Active Grant

• Current Assignee: Veritas Technologies, LLC (Whitehouse Group Ltd.)
• Original Assignee: Symantec Operating Corporation (NortonLifeLock Inc.)
• Inventors: Rao, Santosh S., Dhavale, Poonam P., Sharma, Gopal
• Primary Examiner(s): Flynn; Nathan
• Assistant Examiner(s): WASEL, MOHAMED A
• Application Number:
  US10/627,385
• Time in Patent Office: 2,517 Days
• Field of Search: 709/205, 709/220, 709/232, 714/4, 714/6, 714/13
• US Class Current: 714/4.1
• CPC Class Codes: G06F 11/0709 in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
• CPC Class Codes: G06F 11/0793 Remedial or corrective actions recovery from an exception in an instruction pipeline G06F9/3861; by retry G06F11/1402; for recovering from a failure of a protocol instance or entity H04L69/40
• CPC Class Codes: G06F 11/1425 by reconfiguration of node membership

Abstract

Systems, methods, apparatus and software can configure, support, and make use of a coordinator virtual device to determine which node or nodes of a cluster should be ejected from the cluster as a result of a cluster partition or other error event. Fencing software operating on the cluster nodes monitors the cluster for a cluster partition (split-brain) event, and when such an event occurs, software on the nodes attempts to gain control of the coordinator virtual device. A node that succeeds in gaining control of the coordinator virtual device survives. Nodes failing to gain control of the coordinator virtual device remove themselves or are removed from the cluster. The coordinator virtual device can be established by a virtual device configuration server which provides coordinator virtual device access to cluster nodes acting as virtual device configuration clients.

Claims

1. A method comprising:

providing a coordinator virtual device corresponding to a portion of a physical data storage device;

detecting when a computer system cluster, including a plurality of nodes, is partitioned;

a first node of the plurality of nodes engaging in a race with a second node of the plurality of nodes to gain control of the coordinator virtual device; and

removing the first node of the plurality of nodes from the computer system cluster in response to the first node failing to gain control of the coordinator virtual device by losing the race, wherein

the removing comprises disabling the first node from accessing the portion of the physical data storage device.

2. The method of claim 1 wherein the providing the coordinator virtual device corresponding to the portion of the physical data storage device further comprises:

selecting the portion of the physical data storage device;

associating a physical description of the portion of the physical data storage device with a coordinator virtual device identifier; and

allowing at least one of the plurality of nodes of the computer cluster to access the portion of the physical data storage device via the coordinator virtual device identifier.

3. The method of claim 1 wherein the providing the coordinator virtual device corresponding to the portion of the physical data storage device is performed by at least one virtual device configuration server.

4. The method of claim 3 wherein the at least one virtual device configuration server is separate from the plurality of nodes of the computer cluster and wherein at least one of the plurality of nodes of the computer cluster further comprises a virtual device configuration client.

5. The method of claim 1 further comprising:

reading cluster membership information from the coordinator virtual device corresponding to the portion of the physical data storage device.

6. The method of claim 1 wherein the detecting when the computer system cluster, including the plurality of nodes, is partitioned further comprises:

reading, as performed by one of the plurality of nodes, cluster membership information from the coordinator virtual device corresponding to the portion of the physical data storage device; and

determining whether the cluster membership information indicates that the one of the plurality of nodes is a current member of the computer system cluster.

7. The method of claim 1 further comprising:

writing cluster membership information to the coordinator virtual device corresponding to the portion of the physical data storage device.

8. The method of claim 1 wherein the coordinator virtual device corresponding to the portion of the physical data storage device further comprises cluster membership information.

9. The method of claim 1 wherein the coordinator virtual device corresponding to the portion of the physical data storage device is a coordinator volume.

10. The method of claim 1 wherein the detecting when a computer system cluster is partitioned further comprises:

monitoring a network coupled to each of the plurality of nodes for a heartbeat signal; and

determining when the heartbeat signal is not present for a specified period of time.

11. The method of claim 1 encoded in a computer readable storage medium as instructions executable on a processor, the computer readable storage medium being one of an electronic storage medium, a magnetic storage medium, and an optical storage medium.

12. The method of claim 1 further comprising:

allowing at least one of the plurality of nodes of the computer cluster to exclusively access the portion of the physical data storage device.

13. A system comprising:

a first data storage device;

a virtual device configuration server coupled to the first storage device and including a first memory and a first processor configured to provide a coordinator virtual device corresponding to a portion of the first data storage device;

a plurality of virtual device configuration clients configured as a computer system cluster, a first of the plurality of virtual device configuration clients including a second memory and a second processor configured to;

detect when the computer system cluster is partitioned,

engage in a race with a second of the plurality of virtual device configuration clients to gain control of the coordinator virtual device corresponding to the portion of the first data storage device, and

disable the first of the plurality of virtual device configuration clients from accessing the portion of the first data storage device by removing the first of the plurality of virtual device configuration clients from the computer system cluster in response to the first of the plurality of virtual device configuration clients failing to gain control of the coordinator virtual device by losing the race.

14. The system of claim 13 wherein the first data storage device is at least one of a disk drive, a JBOD, a disk array, and an integrated circuit.

15. The system of claim 13 wherein the first data storage device is coupled to the virtual device configuration server via a network.

16. The system of claim 13 wherein the virtual device configuration server is a volume server, wherein the coordinator virtual device is a coordinator volume, and the plurality of virtual device configuration clients is a plurality of volume clients.

17. The system of claim 13 wherein the first of the plurality of virtual device configuration clients is further configured to read cluster membership information from the coordinator virtual device corresponding to the portion of the first data storage device.

18. The system of claim 17 wherein the first of the plurality of virtual device configuration clients is further configured to determine whether the cluster membership information indicates that the first of the plurality of virtual device configuration clients is a current member of the computer system cluster.

19. The system of claim 13 wherein the first of the plurality of virtual device configuration clients is further configured to write cluster membership information to the coordinator virtual device corresponding to the portion of the first data storage device.

20. The system of claim 13 wherein the coordinator virtual device corresponding to at least a portion of the first data storage device further comprises cluster membership information.

21. The system of claim 13 wherein the first memory and the virtual device configuration server belong to at least one of a host computer system, a cluster node, a storage appliance, a network appliance, and a storage area network (SAN) switch.

22. An apparatus comprising:

a means for providing a coordinator virtual device corresponding to portion of a physical data storage device;

a means for detecting when a computer system cluster, including a plurality of nodes, is partitioned;

a means for engaging a first node of the plurality of nodes in a race with a second node of the plurality of nodes to gain control of the coordinator virtual device; and

a means for disabling the first node from accessing the portion of the first data storage device by removing a first node of the plurality of nodes from the computer system cluster in response to the first node failing to gain control of the coordinator virtual device by losing the race.

23. The apparatus of claim 22 further comprising:

a means for reading cluster membership information from the coordinator virtual device corresponding to the portion of the physical data storage device.

24. The apparatus of claim 22 further comprising:

a means for writing cluster membership information to the coordinator virtual device corresponding to the portion of the physical data storage device.

25. The apparatus of claim 22 further comprising:

a means for determining whether cluster membership information stored in the coordinator virtual device corresponding to the portion of the physical data storage device indicates that the first node of the plurality of nodes is a current member of the computer system cluster.