Method and system for high reliability cluster management

• US 6,856,591 B1
• Filed: 12/15/2000
• Issued: 02/15/2005
• Est. Priority Date: 12/15/2000
• Status: Active Grant

• Current Assignee: Cisco Technology, Inc. (Cisco Systems, Inc.)
• Original Assignee: Cisco Technology, Inc. (Cisco Systems, Inc.)
• Inventors: Duvvury, Murali, Ma, Lucern K.
• Primary Examiner(s): Vanderpuye, Kenneth
• Assistant Examiner(s): Wilson, Robert W.
• Application Number:
  US09/738,409
• Time in Patent Office: 1,523 Days
• Field of Search: 370/216, 370/217, 370/218, 370/219, 370/220, 370/401, 370/408, 709/238
• US Class Current: 370/216
• CPC Class Codes: H04L 41/0213 Standardised network management protocols, e.g. simple network management protocol [SNMP]
• CPC Class Codes: H04L 41/0253 using browsers or web-pages for accessing management information
• CPC Class Codes: H04L 41/0631 using root cause analysis; using analysis of correlation between notifications, alarms or events based on decision criteria, e.g. hierarchy, tree or time analysis
• CPC Class Codes: H04L 41/0816 the condition being an adaptation, e.g. in response to network events
• CPC Class Codes: H04L 41/0836 to enhance reliability, e.g. reduce downtime
• CPC Class Codes: H04L 41/0846 based on copy from other elements
• CPC Class Codes: H04L 41/0897 by horizontal or vertical scaling of resources, or by migrating entities, e.g. virtual resources or entities
• CPC Class Codes: H04L 43/0811 by checking connectivity

Abstract

A method provides high reliability to management of a cluster of network devices. The cluster including a command network device and at least one member network device. The command network device has a commander IP address and a commander MAC address. The method includes: defining a standby group by assigning a virtual IP address and a virtual MAC address; selecting a first network device; which is the command network device, for the standby group; selecting at least one second network device, which has an IP address, for the standby group; defining a standby priority for each network device in the standby group; and binding the standby group to the cluster, by replacing the commander IP address with the virtual IP address and replacing the commander MAC address with the virtual MAC address. Upon the binding, the network device with a highest priority becomes an active commander, and the network device with a second highest priority becomes a standby commander. The the network device with a third highest priority, if any, becomes a passive commander. The cluster is controlled through the active commander using the virtual IP address. The active commander periodically forwards the cluster configuration information to the passive commander. The control of the cluster is switched from the active commander to the standby commander upon a failure of the active commander, the standby commander becoming a current active commander.

Claims

1. A method for providing high reliability to management of a cluster of network devices, said network devices belonging to a management virtual network, said cluster including a command network device and at least one member network device, said command network device having a commander IP address and a commander MAC address, said cluster being managed through said command network device using said commander IP address, said command network device communicating with said member network device using said commander MAC address, said method comprising:

defining a standby group by assigning a virtual IP address and a virtual MAC address to said standby group;

selecting a first network device for said standby group, said first network device being said command network device;

selecting a second network device for said standby group, said second network device having an IP address and belonging to said management virtual network;

defining a standby priority for each network device in said standby group; and

binding said standby group to said cluster, by replacing said commander IP address with said virtual IP address and replacing said commander MAC address with said virtual MAC address, the network device with a highest priority in said standby group becoming an active commander, the network device with a second highest priority in said standby group becoming a standby commander.

2. The method according to claim 1, wherein said defining the standby group further includes assigning a group number and a group name to said standby group.

3. The method according to claim 1, wherein said defining the standby group further includes assigning a group number to said standby group, said virtual MAC address being automatically assigned based on said group number.

4. The method according to claim 1, wherein said second network device is a member of said cluster.

5. The method according to claim 1, further including:

repeating said selecting a second network device so that a desired number of network devices are included in said standby group.

6. The method according to claim 1, wherein in said defining the standby priority, said standby priority is set according to redundant connectivity of the network device in said cluster.

7. The method according to claim 1, wherein in said defining the standby priority, said standby priority is set by a network administrator.

8. The method according to claim 7, wherein said defining the standby priority includes defining said standby priority according to the IP address of each network device of said standby group, when said standby priority for each network device has not been set by the network administrator.

9. The method according to claim 1, wherein said defining the standby priority includes defining said standby priority according to the IP address of each network device of said standby group, when one network device has the same standby priority as another network device in said standby group.

10. The method according to claim 1, further including:

periodically forwarding cluster configuration information of said cluster from said active commander to said standby commander.

11. The method according to claim 10, wherein said cluster configuration information includes a cluster name of said cluster, a MAC address of each member network device, and a member number of each member network device.

12. A method for managing a cluster of network devices, said network devices belonging to a management virtual network, said cluster including a command network device and at least one member network device, said command network device having a commander IP address and a commander MAC address, said method comprising:

defining a standby group for said cluster, by assigning a virtual IP address and a virtual MAC address to said standby group;

selecting a first network device for said standby group, said first network device being said command network device;

selecting a second network device for said standby group, said second network device having an IP address and belonging to said management virtual network;

defining a standby priority for each network device in said standby group;

binding said standby group to said cluster, by replacing the commander IP address with said virtual IP address and replacing the commander MAC address with said virtual MAC address, the network device with a highest priority in said standby group becoming an active commander, the network device with a second highest priority in said standby group becoming a standby commander;

controlling said cluster through said active commander using said virtual IP address; and

switching control of said cluster from said active commander to said standby commander upon a failure of said active commander, said standby commander becoming a current active commander.

13. The method according to claim 12, wherein said switching includes transferring use of said virtual IP address and said virtual MAC address to said standby commander.

14. The method according to claim 12, further comprising:

setting a preemption flag for each network device of said standby group so that said standby priority is maintained whenever the network device becomes a current active commander.

15. The method according to claim 12, further including periodically forwarding cluster configuration information from said active commander to said standby commander.

16. The method according to claim 15, wherein said cluster configuration information includes a cluster name of said cluster, a MAC address of each member network device, and a member number of each member network device.

17. The method according to claim 12, further comprising:

repeating said selecting a second network device so that a desired number of network devices are included in said standby group.

18. The method according to claim 17, wherein in said binding, the network device with a third highest priority in said standby group becomes a passive commander.

19. The method according to claim 18, wherein upon said switching, said passive commander becomes a current standby commander.

20. The method according to claim 19, further comprising:

periodically forwarding said cluster configuration information from said current active commander to said current standby commander.

21. The method according to claim 18, further comprising:

upon a failure of said standby commander, transferring the role of said standby commander to said passive commander, said passive commander becoming a current standby commander.

22. The method according to claim 18, upon a failure of both said active commander and said standby commander at a same time, further comprising:

switching control of said cluster from said active commander to said passive commander, said passive commander becoming a current active commander; and

forwarding cluster member information stored in each member network device to said current active commander.

23. The method according to claim 22, wherein said switching control from said active commander to said passive commander includes transferring use of said virtual IP address and said virtual MAC address.

24. The method according to claim 22, wherein said cluster member information includes a MAC address and a member number of each member network device.

25. The method according to claim 12, further comprising transmitting hello messages at predetermined intervals from said active commander to each network device of said standby group.

26. The method according to claim 12, further comprising transmitting commander heartbeat messages at predetermined intervals from said active commander to each member network device of said cluster.

27. The method according to claim 26, further comprising transmitting member heartbeat messages at predetermined intervals from each member network device of said cluster to said active commander.

28. The method according to claim 27, wherein said member heartbeat messages include current state information and port status information of said member network device.

29. A method for creating a high reliability cluster management system for a cluster of network devices, a port of each network device belonging to a management virtual network, said method comprising:

selecting a first network device having a MAC address,

assigning a cluster IP address and a password to said first network device;

configuring said first network device as command network device for said cluster;

defining a cluster name and SNMP community strings for said cluster, said cluster IP address and said password being used for managing said cluster;

selecting at least one candidate network device for said cluster, a port of said candidate network device belonging to said management virtual network;

adding said at least one candidate network device as a member network device to said cluster;

defining a member number and a CMP address for each member network device, said CMP address used for communication among said member network devices of said cluster;

passing said cluster IP address, said password, and said SNMP strings to each member network device;

defining a standby group by assigning a virtual IP address and a virtual MAC address to said standby group, said standby group including said first network device;

selecting at least one second network device, said second network device having an IP address, a port of said second network device belonging to said management virtual network;

adding said at least one second network device into said standby group;

defining a standby priority for each network device in said standby group;

binding said standby group to said cluster, by replacing said cluster IP address with said virtual IP address and replacing said virtual MAC address with said MAC address of said first network device, the network device with a highest priority in said standby group becoming an active commander, the network device with a second highest priority in said standby group becoming a standby commander; and

periodically forwarding cluster configuration information from said active commander to said standby commander.

30. The method according to claim 29, further comprising:

controlling said cluster through said active commander using said virtual IP address; and

switching control of said cluster from said active commander to said standby commander upon a failure of said active commander, said standby commander becoming a current active commander.

31. A system for managing a cluster of network devices, said network devices belonging to a management virtual network, said cluster including a command network device and at least one member network device, said command network device having a commander IP address and a commander MAC address, said system comprising:

means for defining a standby group for said cluster, by assigning a virtual IP address and a virtual MAC address to said standby group;

means for selecting a first network device for said standby group, said first network device being said command network device;

means for selecting a second network device for said standby group, said second network device having an IP address and belonging to said management virtual network;

means for defining a standby priority for each network device in said standby group;

means for binding said standby group to said cluster, by replacing the commander IP address with said virtual IP address and replacing the commander MAC address with said virtual MAC address, said means for binding including means for making an active commander out of the network device having a highest priority in said standby group, and for making a standby commander out of the network device having a second highest priority in said standby group;

means for controlling said cluster through said active commander using said virtual IP address; and

means for witching control of said cluster from said active commander to said standby commander upon a failure of said active commander, so as to allow said standby commander to become a current active commander.

32. The system according to claim 31, wherein said means for switching includes means for transferring use of said virtual IP address and said virtual MAC address to said standby commander.

33. The system according to claim 31, further comprising:

means for setting a preemption flag for each network device of said standby group so that said standby priority is maintained whenever the network device becomes a current active commander.

34. The system according to claim 31, further including means for periodically forwarding cluster configuration information from said active commander to said standby commander.

35. The system according to claim 34, wherein said cluster configuration information includes a cluster name of said cluster, a MAC address of each member network device, and a member number of each member network device.

36. The system according to claim 31, further comprising:

means for repeating said selecting a second network device so that a desired number of network devices are included in said standby group.

37. The system according to claim 36, wherein said means for making further makes a passive commander out of the network device having a third highest priority in said standby group.

38. The system according to claim 37, wherein said means for switching includes means for transferring the role of said standby commander to said passive commander when said standby commander has become said current active commander, so as to allow said passive commander to become a current standby commander.

39. The system according to claim 38, further comprising:

means for periodically forwarding said cluster configuration information from said current active commander to said current standby commander.

40. The system according to claim 37, further comprising:

means for transferring the role of said standby commander to said passive commander upon a failure of said standby commander, so as to allow said passive commander to become a current standby commander.

41. The system according to claim 37, further comprising:

a second means for switching control of said cluster from said active commander to said passive commander upon a failure of both said active commander and said standby commander at a same time, so as to allow said passive commander to become a current active commander; and

means for forwarding cluster member information stored in each member network device to said current active commander.

42. The system according to claim 41, wherein said second means for switching includes means for transferring use of said virtual IP address and said virtual MAC address.

43. The system according to claim 41, wherein said cluster member information includes a MAC address and a member number of each member network device.

44. The system according to claim 31, further comprising means for transmitting hello messages at predetermined intervals from said active commander to each network device of said standby group.

45. The system according to claim 31, further comprising means for transmitting commander heartbeat messages at predetermined intervals from said active commander to each member network device of said cluster.

46. The system according to claim 45, further comprising means for transmitting member heartbeat messages at predetermined intervals from each member network device of said cluster to said active commander.

47. The system according to claim 46, wherein said member heartbeat messages include current state information and port status information of said member network device.

48. A management system for a cluster of network devices, said network devices belonging to a management virtual network, said system comprising:

a cluster of network devices including a command network device and member network devices, said command network device having an IP address;

a standby group defined by a virtual IP address and a virtual MAC address assigned thereto, said standby group having group member network devices including said command network device, each of group member network devices having a standby priority, a group member network device with a highest standby priority being an active commander, a group member network device with a second highest standby priority being a standby commander, said virtual IP address and said virtual MAC address belonging to said active commander;

a network management station coupled with said cluster and said standby group, said virtual IP address being used for managing said cluster through said active commander; and

a standby group controller for switching control of said cluster from said active commander to said standby commander upon a failure of said active commander, said standby commander becoming a current active commander.

49. The system according to claim 48, wherein said standby group controller transfers use of said virtual IP address and said virtual MAC address upon switching the control of said cluster.

50. The system according to claim 48, wherein said standby group controller includes a cluster information forwarder for periodically forwarding cluster configuration information from said active commander to said standby commander.

51. The system according to claim 48, wherein said standby group includes a passive commander having a third highest standby priority.

52. The system according to claim 51, wherein said standby group controller transferring the role of said standby commander to said passive commander upon a failure of said standby commander, said passive commander becoming a current standby commander.

53. The system according to claim 51, wherein said standby group controller switching control of said cluster from said active commander to said passive commander upon a failure of both said active commander and said standby commander at a same time, said passive commander becoming a current active commander.

54. The system according to claim 53, further comprising a member information forwarder for forwarding cluster member information from said member network devices to said passive commander when said passive commander becomes said current active commander.

55. The system according to claim 48;

wherein said group member network devices of said standby group and said member network devices of said cluster are LAN switches.

56. An apparatus for providing high reliability to management of a cluster of network devices, said apparatus comprising:

a memory, said memory including

a location allocated to store an IP address and a MAC address assigned to said apparatus,

a location allocated to store a virtual IP address and a virtual MAC address, said virtual IP address and said virtual MAC address assigned to and defining a standby group provided for said cluster, and

a location allocated to store a standby priority assigned to said apparatus;

a state machine having an active state and a standby state;

a port allocated to belong to a management virtual network, capable of communicating with network devices of said cluster, and capable of exchanging data packets and messages with network devices of said standby group, said messages including hello messages, said hello messages including state and priority information;

an event determiner for determining an event based on receipt of said hello messages and said state and priority information of received hello messages;

a state controller for changing the state of said state machine in response to a specific event determined by said event determiner;

a commander activator responsive to state transition of said state machine, said commander activator replacing said IP address assigned to said apparatus with said virtual IP address to be used as a source IP address of said apparatus, and replacing said MAC address assigned to said apparatus with said virtual MAC address to be used as a source MAC address of said apparatus; and

a database module capable of maintaining and transmitting cluster configuration information of said cluster when said state machine is in said active state, and capable of receiving and storing cluster configuration information when said state machine is in said standby state.

57. The apparatus according to claim 56, wherein said commander activator operates when said state machine changes into said active state.

58. The apparatus according to claim 56, further comprising a hello message sender for sending out hello messages in predetermined intervals when said state machine is in said active state.

59. The apparatus according to claim 56, further comprising an address filter for receiving data packets and messages having said virtual IP address as a destination IP address only when said state machine is in said active state.

60. The apparatus according to claim 56, further comprising an address filter for receiving data packets and messages having said virtual MAC address as a destination MAC address only when said state machine is in said active state.

61. The apparatus according to claim 56, wherein said state machine further includes a passive state.

62. The apparatus according to claim 61, wherein said database module is further capable of receiving cluster member information from each member network device of said cluster, and of reconstructing said cluster configuration information from the received cluster member information when said state machine changes from said passive state to the active state.

63. The apparatus according to claim 61, wherein said event determiner includes a priority determiner for comparing the priority information of received hello messages with said standby priority of the apparatus.

64. The apparatus according to claim 63, wherein

said event determiner determines an event when hello messages having an active state and a higher priority are not received for a predetermined period of time, and

said state controller changes said state machine from said standby state into said active state, if said event happens when said state machine is in said standby state.

65. The apparatus according to claim 63, wherein

said event determiner determines an event when hello messages having a standby state and a higher priority are not received for a predetermined period to time, and

said state controller changes said state machine from said passive state into said standby state, if said event happens when said state machine is in said passive state.

66. The apparatus according to claim 63, wherein

said event determiner determines an event when both hello messages having an active state and a higher priority and hello messages having a standby state and a higher priority are not received for a predetermined period to time, and

said state controller changes said state machine from said passive state into said active state, if said event happens when said state machine is in said passive state.

67. A program storage device readable by a machine, tangibly embodying a program of instructions executable by the machine to perform a method for providing high reliability to management of a cluster of network devices, said network devices belonging to a management virtual network, said cluster including a command network device and at least one member network device, said command network device having a commander IP address and a commander MAC address, said cluster being managed through said command network device using said commander IP address, said command network device communicating with said member network device using said commander MAC address, said method comprising:

defining a standby group by assigning a virtual IP address and a virtual MAC address to said standby group;

selecting a first network device for said standby group, said first network device being said command network device;

selecting a second network device for said standby group, said second network device having an IP address and belonging to said management virtual network;

defining a standby priority for each network device in said standby group; and

binding said standby group to said cluster, by replacing said commander IP address with said virtual IP address and replacing said commander MAC address with said virtual MAC address, the network device with a highest priority in said standby group becoming an active commander, the network device with a second highest priority in said standby group becoming a standby commander.

68. A program storage device readable by a machine, tangibly embodying a program of instructions executable by the machine to perform a method for managing a cluster of network devices, said network devices belonging to a management virtual network, said cluster including a command network device and at least one member network device, said command network device having a commander IP address and a commander MAC address, said method comprising:

defining a standby group for said cluster, by assigning a virtual IP address and a virtual MAC address to said standby group;

selecting a first network device for said standby group, said first network device being said command network device;

selecting a second network device for said standby group, said second network device having an IP address and belonging to said management virtual network;

defining a standby priority for each network device in said standby group;

binding said standby group to said cluster, by replacing the commander IP address with said virtual IP address and replacing the commander MAC address with said virtual MAC address, the network device with a highest priority in said standby group becoming an active commander, the network device with a second highest priority in said standby group becoming a standby commander;

controlling said cluster through said active commander using said virtual IP address; and

switching control of said cluster from said active commander to said standby commander upon a failure of said active commander, said standby commander becoming a current active commander.

69. A program storage device readable by a machine, tangibly embodying a program of instructions executable by the machine to perform a method for creating a high reliability cluster management system for a cluster of network devices, a port of each network device belonging to a management virtual network, said method comprising:

selecting a first network device having a MAC address,

assigning a cluster IP address and a password to said first network device;

configuring said first network device as command network device for said cluster;

defining a cluster name and SNMP community strings for said cluster, said cluster IP address and said password being used for managing said cluster;

discovering candidate network devices for said cluster, a port of said candidate network devices belonging to said management virtual network;

adding at least one candidate network device as a member network device to said cluster;

defining a member number and a CMP address for each member network device, said CMP address used for communication among said member network devices of said cluster;

passing said cluster IP address, said password, and said SNMP strings to each member network device;

defining a standby group by assigning a virtual IP address and a virtual MAC address to said standby group, said standby group including said first network device;

selecting at least one second network device, said second network device having an IP address, a port of said second network device belonging to said management virtual network;

adding said at least one second network device into said standby group;

defining a standby priority for each network device in said standby group;

binding said standby group to said cluster, by replacing said cluster IP address with said virtual IP address and replacing said virtual MAC address with said MAC address of said first network device, the network device with a highest priority in said standby group becoming an active commander, the network device with a second highest priority in said standby group becoming a standby commander; and

periodically forwarding cluster configuration information from said active commander to said standby commander.