Dynamic recovery from a split-brain failure in edge nodes

• US 10,645,204 B2
• Filed: 02/07/2019
• Issued: 05/05/2020
• Est. Priority Date: 12/21/2016
• Status: Active Grant

• Current Assignee: Nicira Incorporated (Broadcom, Inc.)
• Original Assignee: Nicira Incorporated (Broadcom, Inc.)
• Inventors: Dubey, Ankur, Chandrashekhar, Ganesan, Ravinoothala, Sreeram
• Primary Examiner(s): Jeong, Moo
• Application Number:
  US16/270,580
• Publication Number:
  US 20190173982A1
• Time in Patent Office: 453 Days
• CPC Class Codes: G06F 11/1484 involving virtual machines
• CPC Class Codes: G06F 11/202 where processing functionality is redundant redundant communication control functionality G06F11/2005, redundant storage control functionality G06F11/2089
• CPC Class Codes: G06F 11/2025 using centralised failover control functionality
• CPC Class Codes: G06F 11/2028 eliminating a faulty processor or activating a spare
• CPC Class Codes: G06F 11/2033 switching over of hardware resources
• CPC Class Codes: H04L 41/0661 by reconfiguring faulty entities
• CPC Class Codes: H04L 41/0668 by dynamic selection of recovery network elements, e.g. replacement by the most appropriate element after failure
• CPC Class Codes: H04L 43/0817 by checking functioning
• CPC Class Codes: H04L 43/10 Active monitoring, e.g. heartbeat, ping or trace-route
• CPC Class Codes: H04L 45/28 using route fault recovery
• CPC Class Codes: H04L 5/0055 Physical resource allocation for ACK/NACK for physical mapping arrangements in ARQ protocols H04L1/1861
• CPC Class Codes: H04L 67/62 Establishing a time schedule for servicing the requests
• CPC Class Codes: H04L 69/325 in the network layer [OSI layer 3], e.g. X.25 H04L69/16 takes precedence
• CPC Class Codes: H04L 69/40 for recovering from a failure of a protocol instance or entity, e.g. service redundancy protocols, protocol state redundancy or protocol service redirection management of faults, events, alarms or notifications in data switching networks H04L41/06

Abstract

Some embodiments provide a method for employing the management and control system of a network to dynamically recover from a split-brain condition in the edge nodes of the network. The method of some embodiments takes a corrective action to automatically recover from a split-brain failure occurred at a pair of high availability (HA) edge nodes of the network. The HA edge nodes include an active machine and a standby machine. The active edge node actively passes through the network traffic (e.g., north-south traffic for a logical network), while the standby edge node is synchronized and ready to transition to the active state, should a failure occur. Both HA nodes share the same configuration settings and only one is active until a path, link, or system failure occurs. The active edge node also provides stateful services (e.g., stateful firewall, load balancing, etc.) to the data compute nodes of the network.

Claims

1. A non-transitory machine readable medium of a first edge node of a network storing a program which when executed by at least one processing unit of the edge node determines whether the first edge node should be an active edge node or a standby edge node, the program comprising sets of instructions for:

sending a first message to a controller cluster of the network in response to the first edge node transitioning from a standby state to an active state;

after sending the first message, receiving, from the controller cluster, a second message that identifies a state of the controller cluster;

receiving, from the controller cluster, a third message that identifies a state of a second edge node of the network;

determining, based on the received second and third messages, that the first edge node should not be an active edge node; and

changing a state of the first edge node to standby from active, in response to the determination that the first edge node should not be an active edge node.

2. The non-transitory machine readable medium of claim 1, wherein the first edge node transitions from the standby state to the active state when the first edge node is unable to communicate with the second edge node.

3. The non-transitory machine readable medium of claim 1, wherein the third message identifies the state of the second edge node to be active.

4. The non-transitory machine readable medium of claim 1, wherein the second message indicates whether a majority number of controllers in the controller cluster are healthy and active.

5. The non-transitory machine readable medium of claim 4, wherein the program further comprises a set of instructions for, when the majority number of controllers are not active, changing the state of the first edge node to standby irrespective of the state of the second edge node.

6. The non-transitory machine readable medium of claim 1, wherein the second edge node comprises a primary edge node and the first edge node comprises a secondary edge node in a pair of high availability (HA) edge nodes, wherein the primary edge nodes forwards north-south traffic for the network and the secondary edge node takes over forwarding of traffic when the primary edge node becomes unavailable.

7. The non-transitory machine readable medium of claim 6, wherein the forwarding of traffic comprises performing layer three routing of network traffic to connect the network to one or more external networks.

8. The non-transitory machine readable medium of claim 1, wherein the first and second edge nodes comprise virtual machines that execute on two separate host machines.

9. The non-transitory machine readable medium of claim 1, wherein the first and second edge nodes communicate with each other through a set of private links in order to monitor states of each other.

10. A method for determining whether a first edge node of a network should be an active edge node or a standby edge node, the method comprising:

sending a first message to a controller cluster of the network in response to the first edge node transitioning from a standby state to an active state;

after sending the first message, receiving, from the controller cluster, a second message that identifies a state of the controller cluster;

receiving, from the controller cluster, a third message that identifies a state of a second edge node of the network;

determining, based on the received second and third messages, that the first edge node should not be an active edge node; and

changing a state of the first edge node to standby from active, in response to the determining that the first edge node should not to be an active edge node.

11. The method of claim 10, wherein the first edge node transitions from the standby state to the active state when the first edge node is unable to communicate with the second edge node.

12. The method of claim 10, wherein the third message identifies the state of the second edge node to be active.

13. The method of claim 10, wherein the second message indicates whether a majority number of controllers in the controller cluster are healthy and active.

14. The method of claim 13 further comprising changing the state of the first edge node to standby irrespective of the state of the second edge node when the majority numbers of controllers are not active.

15. The method of claim 10, wherein the second edge node comprises a primary edge node and the first edge node comprises a secondary edge node in a pair of high availability (HA) edge nodes, wherein the primary edge nodes forwards north-south traffic for the network and the secondary edge node takes over the forwarding of traffic when the primary edge node becomes unavailable.

16. The method of claim 10, wherein the first and second edge nodes communicate with each other through a set of private links in order to monitor states of each other.