Method for hierarchical cluster voting in a cluster spreading more than one site

• US 9,015,518 B1
• Filed: 07/11/2012
• Issued: 04/21/2015
• Est. Priority Date: 07/11/2012
• Status: Active Grant

• Current Assignee: NetApp, Inc.
• Original Assignee: NetApp, Inc.
• Inventors: Wenzel, Thomas R.
• Primary Examiner(s): MASKULINSKI, MICHAEL C
• Application Number:
  US13/546,513
• Time in Patent Office: 1,014 Days
• Field of Search: 714/4.1
• US Class Current: 714/4.1
• CPC Class Codes: G06F 11/1425 by reconfiguration of node membership
• CPC Class Codes: G06F 11/1479 Generic software techniques for error detection or fault masking
• CPC Class Codes: G06F 11/187 Voting techniques

Abstract

Resources may be assigned to nodes within a cluster of nodes and the nodes grouped to form sites. Each of the sites may elect a primary node to handle communications between the site and other sites within the cluster. The primary node of a site may be selected by a node identification number assigned to the node. When a node disconnects from a site, the node may form a temporary site and attempt to reestablish communications with the cluster. If communications are restored with the original site, then the node may rejoin the site. If communications with other sites are restored, then the node may form a new site. The dynamic cluster is tolerant to faults and failures within the nodes, because the nodes have information regarding the entire cluster. A site may remain operational with as few as one remaining node.

Claims

1. A method, comprising:

detecting, by a first node, a lost connection from a second node of a first site of a cluster;

attempting, by the first node, to reestablish communications with the second node;

when reestablishing communications is unsuccessful, establishing, by the first node, a second site of the cluster;

establishing, by the first node, communications with a primary node of a third site of the cluster, different from the first site and the second site; and

communicating, by the first node, with the first site through the primary node of the third site.

2. The method of claim 1, further comprising:

establishing communications with a third node of the first site that is also disconnected from the first site; and

electing a primary node for the second site.

3. The method of claim 2, further comprising:

receiving, by the first node, communications from the first site, the second site, and the third site after the first node is elected as the primary node for the second site; and

calculating a topology of the cluster.

4. The method of claim 2, in which the step of electing the primary node comprises selecting the primary node from the second site based, in part, on a node identification number.

5. The method of claim 1, in which the step of establishing communications comprises:

loading information regarding a state of the third site; and

establishing communications through a communication path identified in the state information.

6. The method of claim 1, in which the first node is connected to a redundant array of independent drives.

7. The method of claim 1, in which the first node, the second node, and the third node form a cluster of a redundant storage device.

8. A computer program product, comprising:

a non-transitory computer readable medium comprising;

code to detect, by a first node, a lost connection from a second node of a first site;

code to attempt to reestablish communications with the second node;

code to establish, when reestablishing communications is unsuccessful, a second site; and

code to establish communications with a primary node of a third site.

9. The computer program product of claim 8, in which the medium further comprises:

code to establish communications with a third node of the first site that is also disconnected from the first site; and

code to elect a primary node for the second site.

10. The computer program product of claim 9, in which the medium further comprises:

code to receive, by the primary node, communications from the first site, the second site, and the third site; and

code to calculate a topology of the cluster.

11. The computer program product of claim 10, in which the medium further comprises code to broadcast the calculated topology.

12. The computer program product of claim 9, in which the medium further comprises code to select the primary node based, in part, on a node identification number.

13. The computer program product of claim 8, in which the medium further comprises code to retrieve data from a redundant array of independent drives.

14. The computer program product of claim 8, in which the first node, the second node, and the third node form a cluster of a redundant storage device.

15. An apparatus, comprising:

a first node of a cluster, comprising;

a memory; and

a processor coupled to the memory, in which the processor is configured;

to detect a lost connection from a second node of a first site;

to attempt to reestablish communications with the second node;

to establish, when reestablishing communications is unsuccessful, a second site; and

to establish communications with a primary node of a third site.

16. The apparatus of claim 15, in which the processor is further configured:

to establish communications with a third node of the first site that is also disconnected from the first site; and

to elect a primary node for the second site.

17. The apparatus of claim 16, in which the processor is further configured:

to receive communications from the first site, the second site, and the third site; and

to calculate a topology of the cluster.

18. The apparatus of claim 17, in which the processor is further configured to broadcast the calculated topology.

19. The apparatus of claim 16, in which the processor is further configured to select the primary node based, in part, on a node identification number.

20. The apparatus of claim 15, in which the first node is connected to a redundant array of independent drives (RAID) coupled to the processor.