Distributed load balancing systems

• US 10,135,737 B2
• Filed: 12/01/2014
• Issued: 11/20/2018
• Est. Priority Date: 09/30/2014
• Status: Active Grant

• Current Assignee: Nicira Incorporated (Broadcom, Inc.)
• Original Assignee: Nicira Incorporated (Broadcom, Inc.)
• Inventors: Jain, Jayant, Sengupta, Anirban, Parthasarathy, Mohan, Sequeira, Allwyn, Maskalik, Serge, Lund, Rick
• Primary Examiner(s): Bengzon, Greg C
• Application Number:
  US14/557,290
• Publication Number:
  US 20160094452A1
• Time in Patent Office: 1,450 Days
• CPC Class Codes: G06F 2009/4557 Distribution of virtual machine instances; Migration and load balancing
• CPC Class Codes: G06F 9/45558 Hypervisor-specific management and integration aspects
• CPC Class Codes: H04L 2101/622 Layer-2 addresses, e.g. medium access control [MAC] addresses
• CPC Class Codes: H04L 45/24 Multipath
• CPC Class Codes: H04L 45/44 Distributed routing
• CPC Class Codes: H04L 47/125 by balancing the load, e.g. traffic engineering
• CPC Class Codes: H04L 47/70 Admission control; Resource allocation
• CPC Class Codes: H04L 61/2521 Translation architectures other than single NAT servers
• CPC Class Codes: H04L 61/5069 for group communication, multicast communication or broadcast communication
• CPC Class Codes: H04L 67/1001 for accessing one among a plurality of replicated servers
• CPC Class Codes: H04L 67/1017 based on a round robin mechanism
• CPC Class Codes: H04L 67/1025 Dynamic adaptation of the criteria on which the server selection is based
• CPC Class Codes: H04L 67/1029 using data related to the state of servers by a load balancer

Abstract

Some embodiments provide a novel method for load balancing data messages that are sent by a source compute node (SCN) to one or more different groups of destination compute nodes (DCNs). In some embodiments, the method deploys a load balancer in the source compute node'"'"'s egress datapath. This load balancer receives each data message sent from the source compute node, and determines whether the data message is addressed to one of the DCN groups for which the load balancer spreads the data traffic to balance the load across (e.g., data traffic directed to) the DCNs in the group. When the received data message is not addressed to one of the load balanced DCN groups, the load balancer forwards the received data message to its addressed destination. On the other hand, when the received data message is addressed to one of load balancer'"'"'s DCN groups, the load balancer identifies a DCN in the addressed DCN group that should receive the data message, and directs the data message to the identified DCN. To direct the data message to the identified DCN, the load balancer in some embodiments changes the destination address (e.g., the destination IP address, destination port, destination MAC address, etc.) in the data message from the address of the identified DCN group to the address (e.g., the destination IP address) of the identified DCN.

Claims

1. A distributed load balancing (DLB) system comprising:

a plurality of host computers on which a plurality of source compute nodes (SCNs) execute;

a software forwarding element (SFE) comprising a plurality of SFE ports executing on each host computer;

a plurality of load balancers executing on each host computer, each load balancer associated with one SFE port connected to an SCN executing on the host computer, each load balancer (i) receiving data messages sent by the SCN connected to the SFE port from the associated SFE port along the egress path of the data messages from the SCN to the SFE on the load balancer'"'"'s host computer, (ii) based on a set of load balancing criteria, distributing the data messages that are addressed to a virtual Internet Protocol (IP) address associated with a destination compute node (DCN) group to the DCNs in the group by changing a destination virtual IP address of the DCN group in the data messages to an IP address of one of the DCNs in the group, and (iii) supplying the data messages back to the SFE port to supply the data messages to the SFE on the load balancer'"'"'s host computer to forward the data messages to the DCNs, wherein each SFE port in the plurality of SFE ports has its own load balancer; and

a set of controllers for configuring the load balancers on the host computers, wherein the set of controllers configures the load balancers by collecting message traffic statistics from the load balancers, aggregating the message traffic statistics, computing adjusted load balancing criteria from the aggregated message traffic statistics, and distributing the adjusted load balancing criteria to the load balancers for the load balancers to adjust how they distribute the data messages among the DCNs.

2. The DLB system of claim 1, wherein the set of controllers includes at least one controller for deploying at least one SCN on the host computers.

3. The DLB system of claim 1, wherein the message traffic statistics from a load balancer comprises data regarding a number of data messages that the load balancer forwarded to each DCN in the DCN group.

4. The DLB system of claim 1, wherein the message traffic statistics from a load balancer comprises data regarding a number of data message flows that the load balancer forwarded to each DCN in the DCN group.

5. The DLB system of claim 1, wherein the distributed adjusted load balancing criteria comprises adjusted weight values that affect how the load balancer distributes the data messages among the DCN.

6. The DLB of claim 5, wherein the weight values include one weight value for each DCN, wherein based on the weight values, the load balancer performs a weighted round robin selection of the DCNs for new data message flows.

7. The DLB system of claim 1, wherein the set of controllers further configures the load balancers by distributing addresses of the DCNs that are part of the DCN group.

8. The DLB system of claim 1, wherein the set of controllers further configures the load balancers by (i) distributing a list of the DCNs that are part of the DCN group, and (ii) when a DCN is to be added or removed from the DCN group, distributing an updated list that adds or removes the added or removed DCN from the DCN group.

9. The DLB system of claim 1, wherein

the controller set is further for deploying the SCNs on the host computer, and

the controller set configures each load balancer as part of the deployment of the load balancer'"'"'s SCN.

10. The DLB system of claim 1, wherein:

the DCN group is a first DCN group; and

at least one load balancer is further for intercepting, on the load balancer'"'"'s host computer, data messages that are sent by at least one SCN to a second DCN group and distributing the data messages among the DCNs of the second group.

11. The DLB system of claim 1, wherein

the DCNs are data compute end nodes or middle box service nodes;

and

the SCNs are virtual machines.

12. The DLB system of claim 1, wherein

each load balancer enforces the load balancing criteria needed to distribute the data traffic from the load balancer'"'"'s associated SCN.

13. The DLB system of claim 1, wherein each SFE port is associated with only one load balancer executing on the host computer.

14. A method for a host computing device that belongs to a group of a plurality of host computers, the method comprising:

executing a set of one or more source compute nodes (SCN) on the host computing device;

executing a plurality of load balancers that implement a distributed load balancer (DLB) with other load balancers on host computers of the group, each load balancer associated with one software forwarding element (SFE) port connected to an SCN executing on the host computer, each load balancer in the set (i) receiving data messages sent by the SCN from the associated SFE port along the egress path of data message from the SCN to an SFE on the load balancer'"'"'s host computing device, (ii) based on a set of load balancing criteria, distributing the data messages that are addressed to a virtual internet protocol (IP) address associated with a destination compute node (DCN) group to the DCNs in the group by changing a destination virtual IP address of the DCN group in the data messages to an IP address of one of the DCNs in the group, and (iii) supplying the data messages back to the SFE port to supply the data messages to the SFE on the load balancer'"'"'s host computer to forward the data messages to the DCNs, wherein each SFE port has its own load balancer; and

receiving configuration data from a set of controllers, the configuration data for configuring each load balancer on the host computing device, wherein the set of controllers collects message traffic statistics from load balancers on the plurality of host computers, aggregates the message traffic statistics, computes adjusted load balancing criteria from the aggregated message traffic statistics, and distributes configuration data comprising adjusted load balancing criteria to the load balancers for the load balancers to adjust how they distribute the data messages among the DCNs.

15. The method of claim 14,

wherein the message traffic statistics collected from load balancers on the plurality of host computers comprises one or more attributes of flows distributed by the load balancers to the DCNs in the DCN group, the flow attributes comprising at least one of a number of data messages distributed to each DCN, a number of data message flows distributed to each DCN, and a size of the data message flows distributed to each DCN.

16. The method of claim 14, wherein the configuration data further comprises addresses of the DCNs that are part of the DCN group.

17. The method of claim 14, wherein the SCNs send the data messages to the virtual IP address associated with the DCN group.

18. A non-transitory machine readable medium storing sets of instructions for execution by at least one processing unit of a host computer that belongs to a group of a plurality of host computers, the sets of instructions for:

executing a set of source compute nodes (SCN);

executing a plurality of load balancers that implement a distributed load balancer (DLB) with other load balancers executing on host computers of the group, each load balancer associated with one software forwarding element (SFE) port connected to an SCN executing on the host computer, each load balancer in the set (i) receiving data messages sent by the SCN from the associated SFE port along the egress path of the data messages from the SCN to an SFE on the load balancer'"'"'s host computing device, (ii) based on a set of load balancing criteria, distributing the data messages that are addressed to a virtual internet protocol (IP) address associated with a destination compute node (DCN) group to the DCNs in the group by changing a destination virtual IP address of the DCN group in the data messages to an IP address of one of the DCNs in the group, and (iii) supplying the data messages back to the SFE port to supply the data messages to the SFE on the load balancer'"'"'s host computer to forward the data messages to the DCNs, wherein each SFE port has its own load balancer; and

receiving, from a set of controllers, configuration data for configuring each load balancer, wherein the set of controllers collects message traffic statistics from the load balancers executing on the host computers of the group, aggregates the message traffic statistics, computes adjusted load balancing criteria from the aggregated message traffic statistics, and distributes configuration data comprising adjusted load balancing criteria to the load balancers for the load balancers to adjust how they distribute the data messages among the DCNs.

19. The non-transitory machine readable medium of claim 18, wherein the message traffic statistics collected from load balancers on the plurality of host computers comprises one or more attributes of flows distributed by the load balancers to different DCNs in the DCN group, the flow attributes comprising at least one of a number of data messages in each flow distributed to each DCN, a number of data message flows distributed to each DCN, and a size of the data message flows distributed to each DCN.

20. The non-transitory machine readable medium of claim 18, wherein the configuration data further comprises addresses of the DCNs that are part of the DCN group.

21. The non-transitory machine readable medium of claim 18, wherein the SCNs send the data messages to the virtual IP address associated with the DCN group.