Systems and methods for resolving split-brain scenarios in computer clusters

• US 8,108,715 B1
• Filed: 07/02/2010
• Issued: 01/31/2012
• Est. Priority Date: 07/02/2010
• Status: Active Grant

• Current Assignee: Veritas Technologies, LLC (Whitehouse Group Ltd.)
• Original Assignee: Symantec Corporation (NortonLifeLock Inc.)
• Inventors: Agarwal, Sandeep
• Primary Examiner(s): MASKULINSKI, MICHAEL C
• Application Number:
  US12/830,285
• Time in Patent Office: 578 Days
• Field of Search: None
• US Class Current: 714/10
• CPC Class Codes: G06F 11/0709 in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
• CPC Class Codes: G06F 11/0715 in a system implementing multitasking multitasking per se G06F9/46
• CPC Class Codes: G06F 11/0793 Remedial or corrective actions recovery from an exception in an instruction pipeline G06F9/3861; by retry G06F11/1402; for recovering from a failure of a protocol instance or entity H04L69/40
• CPC Class Codes: G06F 11/1425 by reconfiguration of node membership

Abstract

A computer-implemented method for resolving split-brain scenarios in computer clusters may include (1) identifying a plurality of nodes within a computer cluster that are configured to collectively perform at least one task, (2) receiving, from a node within the computer cluster, a failure notification that identifies a link-based communication failure experienced by the node that prevents the nodes within the computer cluster from collectively performing the task, and, upon receiving the failure notification, (3) immediately prompting each node within the computer cluster to participate in an arbitration event in order to identify a subset of the nodes that is to assume responsibility for performing the task subsequent to the link-based communication failure. Various other methods, systems, and computer-readable media are also disclosed.

Claims

1. A computer-implemented method for resolving split-brain scenarios in computer clusters, the method being performed by a coordination point server comprising at least one processor, the method comprising:

identifying a plurality of nodes within a computer cluster that are configured to collectively perform at least one task;

receiving, at the coordination point server from a node within the computer cluster, a failure notification that identifies a link-based communication failure experienced by the node that prevents the nodes within the computer cluster from collectively performing the task;

upon receiving the failure notification at the coordination point server, immediately causing the coordination point server to initiate an arbitration event within the computer cluster in order to identify a subset of the nodes that is to assume responsibility for performing the task subsequent to the link-based communication failure;

wherein the coordination point server initiates the arbitration event by immediately prompting each node within the computer cluster to participate in the arbitration event.

2. The method of claim 1, wherein the coordination point server is capable of communicating with each node within the computer cluster subsequent to the link-based communication failure.

3. The method of claim 1, wherein receiving the failure notification comprises receiving the failure notification via a kernel-level, low-latency transport mechanism.

4. The method of claim 1, wherein the link-based communication failure experienced by the node comprises at least one of:

failure of the node'"'"'s network interface controller;

failure of a cable connected to the node'"'"'s network interface controller;

disconnection of a cable connected to the node'"'"'s network interface controller;

failure of a network switch connected to the node'"'"'s network interface controller.

5. The method of claim 1, wherein the link-based communication failure experienced by the node is identified within a carrier-loss notification generated by the node'"'"'s operating system.

6. The method of claim 1, wherein immediately prompting each node within the computer cluster to participate in the arbitration event comprises immediately broadcasting an instruction to participate in the arbitration event to each node within the computer cluster.

7. The method of claim 6, wherein broadcasting the instruction comprises broadcasting the instruction via a kernel-level, low-latency transport mechanism.

8. The method of claim 1, wherein the arbitration event comprises a fencing race.

9. The method of claim 1, further comprising:

identifying a node that is the first to contact the coordination point server subsequent to initiation of the arbitration event;

identifying a subcluster within the computer cluster that includes the node that is the first to contact the coordination point server subsequent to initiation of the arbitration event;

limiting responsibility for performing the task to the subcluster.

10. A system for resolving split-brain scenarios in computer clusters, the system comprising:

an identification module installed on a coordination point server and programmed to identify a plurality of nodes within a computer cluster that are configured to collectively perform at least one task;

an arbitration module installed on the coordination point server and programmed to;

receive, at the coordination point server from a node within the computer cluster, a failure notification that identifies a link-based communication failure experienced by the node that prevents the nodes within the computer cluster from collectively performing the task;

upon receiving the failure notification at the coordination point server, immediately cause the coordination point server to initiate an arbitration event within the computer cluster in order to identify a subset of the nodes that is to assume responsibility for performing the task subsequent to the link-based communication failure;

wherein the arbitration module causes the coordination point server to initiate the arbitration event by immediately prompting each node within the computer cluster to participate in the arbitration event;

at least one processor configured to execute the identification module and the arbitration module.

11. The system of claim 10, wherein the arbitration module is capable of communicating with each node within the computer cluster subsequent to the link-based communication failure.

12. The system of claim 10, wherein the arbitration module is programmed to receive the failure notification via a kernel-level, low-latency transport mechanism.

13. The system of claim 10, wherein the link-based communication failure experienced by the node comprises at least one of:

failure of the node'"'"'s network interface controller;

failure of a cable connected to the node'"'"'s network interface controller;

disconnection of a cable connected to the node'"'"'s network interface controller;

failure of a network switch connected to the node'"'"'s network interface controller.

14. The system of claim 10, wherein the link-based communication failure experienced by the node is identified within a carrier-loss notification generated by the node'"'"'s operating system.

15. The system of claim 10, wherein the arbitration module is programmed to immediately broadcast an instruction to participate in the arbitration event to each node within the computer cluster.

16. The system of claim 15, wherein the arbitration module is programmed to broadcast the instruction via a kernel-level, low-latency transport mechanism.

17. The system of claim 10, wherein the arbitration event comprises a fencing race.

18. The system of claim 10, wherein the arbitration module is programmed to:

identify a node that is the first to contact the coordination point server subsequent to initiation of the arbitration event;

identify a subcluster within the computer cluster that includes the node that is the first to contact the coordination point server subsequent to initiation of the arbitration event;

limiting responsibility for performing the task to the subcluster.

19. A non-transitory computer-readable-storage medium comprising one or more computer-executable instructions that, when executed by at least one processor of a coordination point server, cause the coordination point server to:

identify a plurality of nodes within a computer cluster that are configured to collectively perform at least one task;

receive, at the coordination point server from a node within the computer cluster, a failure notification that identifies a link-based communication failure experienced by the node that prevents the nodes within the computer cluster from collectively performing the task;

upon receiving the failure notification at the coordination point server, immediately cause the coordination point server to initiate an arbitration event within the computer cluster in order to identify a subset of the nodes that is to assume responsibility for performing the task subsequent to the link-based communication failure;

wherein the coordination point server initiates the arbitration event by immediately prompting each node within the computer cluster to participate in the arbitration event.

20. The non-transitory computer-readable-storage medium of claim 19, wherein the one or more computer-executable instructions, when executed by the coordination point server, further cause the coordination point server to:

identify a node that is the first to contact the coordination point server subsequent to initiation of the arbitration event;

identify a subcluster within the computer cluster that includes the node that is the first to contact the coordination point server subsequent to initiation of the arbitration event;

limiting responsibility for performing the task to the subcluster.