Method and apparatus for interacting with a network information base in a distributed network control system with multiple controller instances

• US 9,391,928 B2
• Filed: 08/26/2011
• Issued: 07/12/2016
• Est. Priority Date: 07/06/2010
• Status: Active Grant

• Current Assignee: Nicira Incorporated (Broadcom, Inc.)
• Original Assignee: Nicira Incorporated (Broadcom, Inc.)
• Inventors: Koponen, Teemu, Casado, Martin, Stribling, Jeremy, Gude, Natasha
• Primary Examiner(s): Kim, Hee Soo
• Application Number:
  US13/219,557
• Publication Number:
  US 20130060817A1
• Time in Patent Office: 1,782 Days
• Field of Search: 709/224
• US Class Current: 1/1
• CPC Class Codes: G06F 11/07 Responding to the occurrence of a fault, e.g. fault tolerance
• CPC Class Codes: G06F 15/17312 Routing techniques specific to parallel machines, e.g. wormhole, store and forward, shortest path problem congestion routing on a LAN H04L45/00
• CPC Class Codes: H04L 12/4633 Interconnection of networks using encapsulation techniques, e.g. tunneling
• CPC Class Codes: H04L 2101/622 Layer-2 addresses, e.g. medium access control [MAC] addresses
• CPC Class Codes: H04L 41/0816 the condition being an adaptation, e.g. in response to network events
• CPC Class Codes: H04L 41/0853 by actively collecting configuration information or by backing up configuration information
• CPC Class Codes: H04L 41/0893 Assignment of logical groups to network elements
• CPC Class Codes: H04L 41/0894 Policy-based network configuration management
• CPC Class Codes: H04L 41/0895 Configuration of virtualised networks or elements, e.g. virtualised network function or OpenFlow elements
• CPC Class Codes: H04L 41/0896 Bandwidth or capacity management, i.e. automatically increasing or decreasing capacities flow or congestion control using dynamic resource allocation, e.g. in-call renegotiation, H04L47/76
• CPC Class Codes: H04L 41/122 of virtualised topologies, e.g. software-defined networks [SDN] or network function virtualisation [NFV]
• CPC Class Codes: H04L 45/00 Routing or path finding of packets in data switching networks routing or path finding in wireless networks H04W40/00
• CPC Class Codes: H04L 45/02 Topology update or discovery
• CPC Class Codes: H04L 45/036 Updating the topology between route computation elements, e.g. between OpenFlow controllers
• CPC Class Codes: H04L 45/586 of virtual routers
• CPC Class Codes: H04L 45/76 Routing in software-defined topologies, e.g. routing between virtual machines
• CPC Class Codes: H04L 47/783 Distributed allocation of resources, e.g. bandwidth brokers
• CPC Class Codes: H04L 49/00 Packet switching elements
• CPC Class Codes: H04L 49/1546 using pipelined operation
• CPC Class Codes: H04L 49/3063 Pipelined operation
• CPC Class Codes: H04L 49/70 : Virtual switches
• CPC Class Codes: H04L 61/5007 : Internet protocol [IP] addresses

Abstract

A control system including several controllers for managing several switching elements. A first controller registers a second controller for receiving a notification when a data tuple changes in a network information base (NIB) storage of the first controller that stores data for managing a set of switching elements. The first controller changes the data tuple in the NIB. The first controller sends the notification to the second controller of the change to the data tuple in the NIB. The first and second controllers operate on two different computing devices. Each controller receives logical control plane data for specifying logical datapath sets and converts the logical control plane data to physical control plane data for enabling the switching elements to implement the logical datapath sets.

Claims

1. A non-transitory machine readable medium for a first controller, the medium storing a program for managing a set of managed forwarding elements, the program comprising sets of instructions for:

storing a set of data in a local network information base (NIB) storage of the first controller, wherein the local NIB storage stores data for managing the set of managed forwarding elements;

registering a second controller for receiving a notification when the set of data changes in the local NIB storage, wherein the first controller is a master of the set of data in the local NIB storage and the second controller is for managing a particular managed forwarding element of the set of managed forwarding elements, wherein the set of managed forwarding elements implements a logical forwarding element, wherein the first and second controllers execute on separate physical machines from the managed forwarding elements managed by the controllers;

changing the set of data in the local NIB storage; and

sending the notification to the second controller of the change to the set of data in the local NIB storage, wherein the second controller propagates a corresponding change to the particular managed forwarding element to modify forwarding behaviors of the particular managed forwarding element based on the corresponding change.

2. The non-transitory machine readable medium of claim 1, wherein the set of managed forwarding elements are for forwarding data between machines that operate on separate computing devices from the managed forwarding elements.

3. The non-transitory machine readable medium of claim 1, wherein each controller (i) receives logical control plane data for specifying logical datapath sets of the logical forwarding element and (ii) converts the logical control plane data to physical control plane data for enabling the managed forwarding elements to implement the logical datapath sets.

4. The non-transitory machine readable medium of claim 3, wherein the propagated corresponding change includes physical control plane data.

5. The non-transitory machine readable medium of claim 4, wherein the particular managed forwarding element converts the physical control plane data to physical forwarding plane data.

6. The non-transitory machine readable medium of claim 1, wherein the program further comprises a set of instructions for detecting when the set of data in the local NIB changes.

7. The non-transitory machine readable medium of claim 1,

wherein the first and second controllers each comprises a secondary storage,

wherein the program further comprises a set of instructions for recording the change in the set of data in the secondary storage of the first controller.

8. The non-transitory machine readable medium of claim 1,

wherein the first and second controllers each comprises a secondary storage,

wherein the set of instructions for sending the notification to the second controller comprises a set of instructions for sending the notification to the secondary storage of the second controller,

wherein, upon detecting receipt of the notification at the second controller'"'"'s secondary storage, the second controller directs the first controller'"'"'s secondary storage to record the change in the set of data.

9. The non-transitory machine readable medium of claim 7, wherein the set of instructions for sending the notification to the second controller comprises a set of instructions for sending the notification to the secondary storage of the second controller after recording the change in the first controller'"'"'s secondary storage.

10. The non-transitory machine readable medium of claim 1, wherein the program further comprises a set of instructions for receiving a request from the second controller to register for notification of a change in the set of data.

11. A method of managing a set of managed forwarding elements of a plurality of managed forwarding elements managed by a plurality of controllers in a network control system, the method comprising:

at a first controller, changing the set of data stored in a local network information base (NIB) storage of the first controller, wherein the local NIB storage stores data for managing the set of managed forwarding elements; and

from the first controller, sending a notification to a secondary storage of a second controller of the change to the set of data in the local NIB storage of the first controller after recording the change in a secondary storage of the first controller, wherein the second controller updates a local NIB storage of the second controller after detecting the change in the secondary storage of the second controller and propagates a corresponding change to a particular managed forwarding element of the set of managed forwarding elements to modify forwarding behaviors of the particular managed forwarding element based on the corresponding change, wherein the first and second controllers execute on separate physical machines from the managed forwarding elements managed by the controllers.

12. The method of claim 11, wherein changing the set of data comprises receiving, from a third controller, a request to modify the set of data stored in the first controller'"'"'s NIB storage.

13. The method of claim 11,

wherein changing the set of data comprises receiving a request to modify the set of data stored in the first controller'"'"'s network information base (NIB) storage,

wherein the request to modify the set of data is generated at the first controller.

14. The method of claim 11,

wherein the set of data includes physical control plane data that the first controller generates from logical datapath set data,

wherein the corresponding change that is propagated to the particular managed forwarding element comprises physical control plane data, wherein the particular managed forwarding element converts the physical control plane data to physical forwarding plane data.

15. The method of claim 11, wherein in response to the notification, the second controller changes the set of data in the second controller'"'"'s NIB storage, in order to have the NIB storages of the first and second controllers contain the same data.

16. The method of claim 11, wherein the s particular managed forwarding element is a first managed forwarding element, wherein the second controller propagates the corresponding change to a second managed forwarding element of the set of managed forwarding elements to modify forwarding behaviors of the second managed forwarding element based on the corresponding change.

17. A distributed network control system comprising:

a first controller computer that executes a first controller instance for execution by at least one processing unit, the first controller instance for managing a first set of data in a first network information base (NIB) storage of the first controller instance regarding a set of managed forwarding elements and making changes to the first set of data in the first NIB storage based on detected changes in the set of managed forwarding elements; and

a second controller computer that executes a second controller instance for execution by at least one processing unit, the second controller instance for managing a second set of data in a second NIB storage of the second controller instance regarding a logical forwarding element that is implemented on the set of managed forwarding elements, wherein the second controller instance registers to receive notifications from the first controller instance when the first set of data in the first NIB storage changes, wherein the first and second controller computers are separate physical machines from the set of managed forwarding elements.

18. The distributed network control system of claim 17, wherein the NIB storage of the first controller instance is at least partially replicated with the NIB storage of the second controller instance, and this replication allows the NIB storage to serve as the communication medium between the first controller instance and the second controller instance.

19. The distributed network control system of claim 18,

wherein the first and second controller instances each comprises a secondary storage,

wherein the secondary storage of the first controller instance further serves as a communication medium with the second controller instance by serving as an intermediary storage when the first controller instance'"'"'s NIB storage is being replicated with the second controller instance'"'"'s NIB storage.

20. The distributed network control system of claim 19, wherein the secondary storages of the first and second controller instances each include a persistent transactional database.

21. The distributed network control system of claim 19, wherein the secondary storages of the first and second controller instances each include a distributed hash table (DHT) instance.

22. The distributed network control system of claim 17, wherein the first controller instance'"'"'s NIB storage is further for receiving requests to modify stored data from a third controller instance.

23. The distributed network control system of claim 17, wherein the first controller instance'"'"'s NIB storage is further for receiving, from a third controller instance, requests for notification when a set of data stored in the NIB storage of the first controller changes.

24. The distributed network control system of claim 17, wherein the first controller instance further comprises a secondary storage that serves as a medium for communication with the second controller instance.

25. The distributed network control system of claim 24, wherein data from the first controller instance'"'"'s NIB storage is at least partially replicated in the secondary storage of the first controller instance.

26. The distributed network control system of claim 17, wherein the first controller instance is further for propagating data to at least the set of managed forwarding elements.