Method to dynamically change cluster or distributed system configuration

• US 6,748,429 B1
• Filed: 01/10/2000
• Issued: 06/08/2004
• Est. Priority Date: 01/10/2000
• Status: Expired due to Term

• Current Assignee: Oracle America, Inc. (Oracle Corporation)
• Original Assignee: Sun Microsystems Incorporated (Oracle Corporation)
• Inventors: Smaalders, Bart, Kondaka, Krishna, Panicker, Sundeep, Talluri, Madhusudhan
• Primary Examiner(s): Wiley, David
• Assistant Examiner(s): Avellino, Joseph E.
• Application Number:
  US09/480,330
• Time in Patent Office: 1,611 Days
• Field of Search: 709/220-221, 709/222, 709/226, 709/229, 714/3-4
• US Class Current: 709/221
• CPC Class Codes: G06F 2209/505 Clust
• CPC Class Codes: G06F 9/5061 Partitioning or combining of resources
• CPC Class Codes: H04L 41/0816 the condition being an adaptation, e.g. in response to network events
• CPC Class Codes: H04L 41/0856 by backing up or archiving configuration information
• CPC Class Codes: H04L 41/0873 Checking configuration conflicts between network elements
• CPC Class Codes: H04L 41/12 Discovery or management of network topologies
• CPC Class Codes: H04L 69/40 for recovering from a failure of a protocol instance or entity, e.g. service redundancy protocols, protocol state redundancy or protocol service redirection management of faults, events, alarms or notifications in data switching networks H04L41/06

Abstract

Method for enabling the dynamic modification of cluster configurations, and apparatus including software to perform the method. To enable this dynamic modification, cluster configuration data is stored as a table in a cluster configuration repository that is accessible from all nodes in the cluster. Accordingly, the present invention enables the modification of the cluster configuration from any node in the cluster dynamically. When a reconfiguration command is given, the configuration table is changed and all the nodes in the cluster are notified of the changed configuration in parallel. Following the notification by the nodes of the changed cluster configuration, the changes to the cluster are implemented dynamically as specified by the command.

Claims

1. A method for dynamically changing the system configuration of a distributed computer system while retaining overall system functionality, the system including a plurality of components and further including a plurality of nodes, the method comprising the steps of:

for each one of the plurality of nodes, storing system configuration data, including component attributes, on at least one storage device accessible by that node and every other node of the plurality of nodes in the distributed computer system;

initiating a command to effect a system configuration changes whereby at least one attribute of at least one component is changed by a first node of the distributed computer system, wherein the first node operates to change the system configuration data by accessing the at least one storage device to change the at least one attribute of the at least one component;

notifying a second node of the distributed computer system of a system configuration change;

accessing the system configuration data, by the second node of the plurality of nodes in the distributed computer system, to initiate changing only the at least one attribute of the at least one system component affected by the system configuration change; and

wherein the first and second nodes are not the same node.

2. The method of claim 1, each node further including memory, the distributed computing system configured in accordance with the system configuration data, whereby the changing step comprises, at each one of the plurality of nodes, the further steps of:

responsive to the initiating step, copying the system configuration data from the storage device to the memory as an in-memory copy of the system configuration data;

making changes to the in-memory copy of the system configuration data reflecting the system configuration change; and

using the in-memory copy, changing only the at least one attribute of the at least one system component affected by the system configuration change.

3. The method of claim 2 wherein the copying step comprises further steps of:

making a backup copy of the in-memory copy; and

making changes to the backup copy reflecting the system configuration change, the backup copy thereby defining the proposed system configuration change.

4. The method of claim 3, wherein the step of changing only the at least one attribute of the at least one system component comprises the further step of:

calculating, from the backup copy, the at least one attribute needed to complete the system configuration change on the at least one affected component.

5. The method of claim 4, wherein the step of changing only the at least one attribute of the at least one system component comprises the further steps of:

notifying a topology manager of the system configuration change;

creating, with the topology manager, a proposed cluster topology graph reflecting the system configuration change;

comparing the topology graph with a previous version of the cluster topology graph;

determining the differences between the proposed topology graph and the previous version of the topology graph, the differences defining action items to effect the system configuration change; and

executing the action items, thereby implementing the system configuration change.

6. The method of claim 5 wherein the executing step further comprises the steps of:

notifying a path manager of changes to paths and adapters occasioned by the system configuration change; and

calling, with the path manager, the affected clients to effect the system configuration change.

7. The method of claim 1 wherein the step of storing system configuration data comprises the further step of storing the system configuration data in a cluster configuration repository on the storage device.

8. The method of claim 7 comprising the further step of locking the cluster configuration repository during at least a portion of the changing step.

9. A computer readable storage medium storing instructions that, when read and executed by a computer, cause the computer to perform a method for dynamically changing the system configuration of a distributed computer system while retaining overall system functionality, the system including a plurality of components further including a plurality of nodes, the method comprising the steps of:

for each one of the plurality of nodes, storing system configuration data on at least one storage device accessible by the node and every other node of the plurality of nodes in the distributed computer system;

initiating a command to effect a system configuration change whereby at least one attribute of at least one component is changed by a first node of the distributed computer system, wherein the first node operates to change the system configuration data by accessing the at least one storage device to change the at least one attribute of the at least one component;

notifying a second node of the distributed computer system of a system configuration change;

accessing the system configuration data, by a second node of the plurality of nodes in the distributed computer system, to initiate, changing only the at least one attribute of the at least one system component affected by the system configuration change; and

wherein the first and second nodes are not the same node.

10. The computer readable storage medium of claim 9 wherein each node further includes memory and access to a storage device, the system defined by the system configuration data, whereby the changing step comprises, at each one of the plurality of nodes, the further steps of:

responsive to the initiating step, at each one of the plurality of nodes copying the system configuration data from the storage device to the memory as an in-memory copy of the system configuration data;

at each one of the plurality of nodes, making changes to the in-memory copy of the system configuration data reflecting the system configuration change; and

using the in-memory copy, at each one of the plurality of nodes, changing only those system components affected by the system configuration change.

11. The computer readable storage medium of claim 10 wherein the copying step comprises further steps of:

making a backup copy of the in-memory copy; and

making changes to the backup copy reflecting the system configuration change, the backup copy thereby defining the proposed system configuration change.

12. The computer readable storage medium of claim 11, wherein the step of changing only the at least one attribute of the at least one system component comprises the further step of:

calculating, from the backup copy, the at least one attribute needed to complete the system configuration change on the at least one affected component.

13. The computer readable storage medium of claim 12, wherein the step of changing only the at least one attribute of the at least one system component comprises the further steps of:

notifying a topology manager of the system configuration change;

creating, with the topology manager, a proposed cluster topology graph reflecting the system configuration change;

comparing the topology graph with a previous version of the cluster topology graph;

determining the differences between the proposed topology graph and the previous version of the topology graph, the differences defining action items to effect the system configuration change; and

executing the action items, thereby changing the cluster topology graph.

14. The computer readable storage medium of claim 13 wherein the executing step further comprises the steps of:

notifying a path manager of changes to paths and adapters occasioned by the system configuration change; and

calling, with the path manager, the affected clients to effect the system configuration change.

15. The computer readable storage medium of claim 9 wherein the step of storing system configuration data comprises the further step of storing the system configuration data in a cluster configuration repository on the storage device.

16. The computer readable storage medium of claim 15, comprising the further step of locking the cluster configuration repository during at least a portion of the changing step.

17. Apparatus for dynamically changing the system configuration of a distributed computer system while retaining overall system functionality, the system including a plurality of components further including a plurality of nodes, the apparatus comprising:

system configuration data stored on at least one storage device accessible by each one of the plurality of nodes;

at least one component having at least one attribute to be changed by a system configuration change; and

a program for;

initiating a command to effect a system configuration change, whereby at least one attribute of at least one component is changed by a first node of the distributed computer system, wherein the first node operates to change the system configuration data by accessing the at least one storage device to change the at least one attribute of the at least one component;

accessing the system configuration data, by a second node of the plurality of nodes in the distributed computer system, to initiate changing only the at least one attribute of the at least one system component affected by the system configuration change; and

wherein the first and second nodes are not the same node.

18. The apparatus of claim 17, each node further including memory and access to a storage device, the system defined by the system configuration data, the apparatus further comprising:

at each one of the plurality of nodes system configuration data copied from the storage device to the memory as an in-memory copy of the system configuration data;

the program further including a process for making changes to the in-memory copy of the system configuration data reflecting the system configuration change, and using the in-memory copy, at each one of the plurality of nodes, for changing only those system components affected by the system configuration change.

19. The apparatus of claim 18 wherein the program further forms a backup copy of the in-memory copy, the program of the apparatus further comprising a mechanism to make changes to the backup copy reflecting the system configuration change, the backup copy thereby defining the proposed system configuration.

20. The apparatus of claim 19, wherein the program calculates, from the backup copy, the at least one attribute needed to complete the system configuration change on the at least one affected component.

21. The apparatus of claim 20, further comprising:

a topology manager which is notified of the system configuration change;

a proposed cluster topology graph created by the topology manager, the proposed topology graph reflecting the system configuration change;

action items required to effect the system configuration change, the action items defined by the differences between the proposed topology graph and a previous version of the topology graph,

whereby the execution of the action items changes the cluster topology graph.

22. The apparatus of claim 21 further comprising a path manager which is notified of changes to paths and adapters occasioned by the system configuration change and which calls the affect path manager clients to effect the system configuration change.

23. The apparatus of claim 17 wherein the system configuration data further comprises system configuration data stored in a cluster configuration repository on the storage device.

24. The apparatus of claim 23 comprising a mutual exclusion mechanism for locking the system cluster configuration repository during at least a portion of the system configuration change process.

25. A method for changing system configuration of a cluster of computing systems, the cluster computing system including a plurality of nodes, the method comprising:

storing system configuration data for the cluster in a cluster configuration repository, wherein the cluster configuration repository is stored on a storage device that is directly accessible by each one the plurality of nodes in the cluster;

accessing the cluster configuration repository stored on the storage device by a first node of the plurality of nodes in response to a need for a system configuration change, wherein the first node directly accesses the storage device in order to access the cluster configuration repository;

modifying the cluster configuration repository by the first node after accessing the cluster configuration repository by the first node;

notifying two or more other nodes of the plurality of nodes of a system configuration change;

accessing the cluster configuration repository stored on the storage device by the two or more other nodes when the two or more other nodes have been notified, wherein the two or more other nodes directly access the storage device in order to access the cluster configuration repository; and

initiating, by the two or more other nodes, one or more actions needed to implement changes after the accessing of the cluster configuration repository.

26. A method as recited in claim 25,

wherein the cluster configuration repository is accessed and effectively changed by a first thread in user space running on the first node; and

wherein a two or more other threads in kernel space running on the two or more other nodes initiate one or more actions needed to implement changes.

27. A method as recited in claim 26, wherein the user first thread is a first topology manager process running in user space and operates to modify the cluster configuration repository.

28. A method as recited in claim 27,

wherein the two or more other threads are topology managers running in kernel space which operates to:

define a series of action items in response to the changes;

execute the action items to effect the changes; and

wherein the method further comprises;

notifying a path manager of changes; and

calling path manager clients to carry out changes.

29. A method as recited in claims 27, wherein the method further comprises:

registering two or more other nodes of the plurality of nodes with a call back function that notifies the two or more other nodes of a change to the cluster configuration repository.