Dynamic recovery from a split-brain failure in edge nodes

• US 10,237,123 B2
• Filed: 12/21/2016
• Issued: 03/19/2019
• Est. Priority Date: 12/21/2016
• Status: Active Grant

• Current Assignee: Nicira Incorporated (Broadcom, Inc.)
• Original Assignee: Nicira Incorporated (Broadcom, Inc.)
• Inventors: Dubey, Ankur, Chandrashekhar, Ganesan, Ravinoothala, Sreeram
• Primary Examiner(s): Yao, Kwang B
• Assistant Examiner(s): Jeong, Moo
• Application Number:
  US15/387,549
• Publication Number:
  US 20180176073A1
• Time in Patent Office: 818 Days
• CPC Class Codes: G06F 11/1484 involving virtual machines
• CPC Class Codes: G06F 11/202 where processing functionality is redundant redundant communication control functionality G06F11/2005, redundant storage control functionality G06F11/2089
• CPC Class Codes: G06F 11/2025 using centralised failover control functionality
• CPC Class Codes: G06F 11/2028 eliminating a faulty processor or activating a spare
• CPC Class Codes: G06F 11/2033 switching over of hardware resources
• CPC Class Codes: H04L 41/0661 by reconfiguring faulty entities
• CPC Class Codes: H04L 41/0668 by dynamic selection of recovery network elements, e.g. replacement by the most appropriate element after failure
• CPC Class Codes: H04L 43/0817 by checking functioning
• CPC Class Codes: H04L 43/10 Active monitoring, e.g. heartbeat, ping or trace-route
• CPC Class Codes: H04L 45/28 using route fault recovery
• CPC Class Codes: H04L 5/0055 Physical resource allocation for ACK/NACK for physical mapping arrangements in ARQ protocols H04L1/1861
• CPC Class Codes: H04L 67/62 Establishing a time schedule for servicing the requests
• CPC Class Codes: H04L 69/325 in the network layer [OSI layer 3], e.g. X.25 H04L69/16 takes precedence
• CPC Class Codes: H04L 69/40 for recovering from a failure of a protocol instance or entity, e.g. service redundancy protocols, protocol state redundancy or protocol service redirection management of faults, events, alarms or notifications in data switching networks H04L41/06

Abstract

Some embodiments provide a method for employing the management and control system of a network to dynamically recover from a split-brain condition in the edge nodes of the network. The method of some embodiments takes a corrective action to automatically recover from a split-brain failure occurred at a pair of high availability (HA) edge nodes of the network. The HA edge nodes include an active machine and a standby machine. The active edge node actively passes through the network traffic (e.g., north-south traffic for a logical network), while the standby edge node is synchronized and ready to transition to the active state, should a failure occur. Both HA nodes share the same configuration settings and only one is active until a path, link, or system failure occurs. The active edge node also provides stateful services (e.g., stateful firewall, load balancing, etc.) to the data compute nodes of the network.

Claims

1. A method for specifying an operational state of first and second logical gateways, the method comprising:

receiving a message from the first gateway that indicates the first gateway has transitioned from a standby state to an active state, the first and second gateways (i) serving as a pair of high availability (HA) logical gateways of a logical network implemented on a physical network and (ii) connecting the logical network to a network external to the logical network;

determining that the second gateway is in the active state when the first gateway is in the active state;

determining that the first and second gateways do not communicate with each other; and

based on the determination that the first and second gateways do not communicate with each other, directing the first gateway to transition back to the standby state.

2. The method of claim 1, wherein the pair of HA gateways forwards north-south traffic for the logical network.

3. The method of claim 2, wherein an active gateway forwards the north-south traffic for the logical network and a standby gateway takes over the forwarding of the north-south traffic when the active gateway becomes unavailable.

4. The method of claim 2, wherein the forwarding of network traffic comprises performing layer three routing of network traffic to connect the logical network to one or more external networks.

5. The method of claim 1, wherein the first and second gateways comprise virtual machines executing on different host machines.

6. The method of claim 1, wherein the first and second gateways communicate with each other through a set of private links in order to monitor states of each other.

7. The method of claim 1, wherein the message is a first message, wherein determining that the first and second gateways do not communicate with each other comprises receiving a second message from the first gateway that indicates the first and second gateways are disconnected.

8. The method of claim 7, wherein the first gateway sends the second message when the first gateway sends a third message to the second gateway and does not receive an acknowledgement back from the second gateway within a specified period.

9. The method of claim 8, wherein the third message comprises a bidirectional forwarding detection (BFD) message.

10. The method of claim 1, wherein the message is a first message, wherein directing the first gateway to transition back to the first state comprises sending a second message to the first gateway instructing the first gateway to move back into a standby state.

11. A non-transitory machine readable medium, storing a program, which when implemented by at least one processing unit specifies an operational state of first and second logical gateways, the program comprising sets of instructions for:

receiving a message from the first gateway that indicates the first gateway has transitioned from a standby state to an active state, the first and second gateways (i) serving as a pair of high availability (HA) logical gateways of a logical network implemented on a physical network and (ii) connecting the logical network to a network external to the logical network;

determining that the second gateway is in the active state when the first gateway is in the active state;

determining that the first and second gateways do not communicate with each other; and

based on the determination that the first and second gateways do not communicate with each other, directing the first gateway to transition back to the standby state.

12. The non-transitory machine readable medium of claim 11, wherein the pair of HA gateways forwards north-south traffic for the logical network.

13. The non-transitory machine readable medium of claim 12, wherein an active gateway forwards the north-south traffic for the logical network and a standby gateway takes over the forwarding of the north-south traffic when the active gateway becomes unavailable.

14. The non-transitory machine readable medium of claim 12, wherein the forwarding of network traffic comprises performing layer three routing of network traffic to connect the logical network to one or more external networks.

15. The non-transitory machine readable medium of claim 11, wherein the first and second gateways comprise virtual machines executing on different host machines.

16. The non-transitory machine readable medium of claim 11, wherein the first and second gateways communicate with each other through a set of private links in order to monitor states of each other.

17. The non-transitory machine readable medium of claim 11, wherein the message is a first message, wherein the set of instructions for determining that the first and second gateways do not communicate with each other comprises a set of instructions for receiving a second message from the first gateway that indicates the first and second gateways are disconnected.

18. The non-transitory machine readable medium of claim 17, wherein the first gateway sends the second message when the first gateway sends a third message to the second gateway and does not receive an acknowledgement back from the second gateway within a specified period.

19. The non-transitory machine readable medium of claim 18, wherein the third message comprises a bidirectional forwarding detection (BFD) message.

20. The non-transitory machine readable medium of claim 11, wherein the message is a first message, wherein the set of instructions for directing the first gateway to transition back to the first state comprises a set of instructions for sending a second message to the first gateway instructing the first gateway to move back into a standby state.