Resource management in a clustered computer system

• US 6,151,688 A
• Filed: 02/14/1998
• Issued: 11/21/2000
• Est. Priority Date: 02/21/1997
• Status: Expired due to Term

• Current Assignee: RPX Corporation
• Original Assignee: Novell Incorporated (Open Text Corporation)
• Inventors: Murphy, David, Wipfel, Robert A
• Primary Examiner(s): Hua, Ly V.
• Assistant Examiner(s): REVAK, CHRISTOPHER A
• Application Number:
  US09/024,011
• Time in Patent Office: 1,011 Days
• Field of Search: 714/5, 714/48, 714/49, 714/718, 714/8, 714/39, 714/42, 714/43, 714/47, 714/55, 714/57, 709/224, 709/223, 709/226, 709/251
• US Class Current: 714/48
• CPC Class Codes: G06F 11/008 Reliability or availability analysis

Abstract

Methods, systems, and devices are provided for managing resources in a computing cluster. The managed resources include cluster nodes themselves, as well as sharable resources such as memory buffers and bandwidth credits that may be used by one or more nodes. Resource management includes detecting failures and possible failures by node software, node hardware, interconnects, and system area network switches and taking steps to compensate for failures and prevent problems such as uncoordinated access to a shared disk. Resource management also includes reallocating sharable resources in response to node failure, demands by application programs, or other events. Specific examples provided include failure detection by remote memory probes, emergency communication through a shared disk, and sharable resource allocation with minimal locking.

Claims

1. A Method for managing resources in a cluster, the method comprising the computer-implemented steps of:

remotely probing memory in a device in the cluster; and

using one of the following to determine whether the probed device has failed;

a counter value retrieved by the remote probing, or the unavailability of such a counter value;

wherein the probing step probes volatile memory in a remote cluster node.

2. The method of claim 1, wherein the probing step and the using step are repeated at predetermined intervals to monitor the remote device.

3. The method of claim 1, wherein the probing step and the using step are performed in preparation for sending at least one of the following to the remote device:

data, or control information.

4. The method of claim 1, wherein the using step determines that the remote device has failed in response to determining that the counter value equals a previously retrieved counter value.

5. The method of claim 1, wherein the using step determines that the remote device has failed in response to the presence of a status value retrieved by the probing step.

6. The method of claim 1, further comprising the step of removing from the cluster a failed node, the node'"'"'s failure being detected by the probing and using steps.

66. A computer storage medium having a configuration that represents data and instructions which will cause at least a portion of a computer system to perform method steps for managing resources in a cluster computing system, the method steps comprising the steps of claim 1.

7. A method for managing resources in a cluster, the method comprising the computer-implemented steps of:

remotely probing memory in a device in the cluster; and

using one of the following to determine whether the probed device has failed;

a counter value retrieved by the remote probing, or the unavailability of such a counter value;

wherein the probing step probes memory in a remote cluster interconnect.

8. The method of claim 7, wherein the probing step and the using step are repeated at predetermined intervals to monitor the remote device.

9. The method of claim 7, wherein the probing step and the using step are performed in preparation for sending at least one of the following to the remote device:

data, or control information.

10. The method of claim 7, wherein the using step determines that the remote device has failed in response to determining that the counter value equals a previously retrieved counter value.

11. The method of claim 7, wherein the using step determines that the remote device has failed in response to the presence of a status value retrieved by the probing step.

12. The method of claim 7, further comprising the step of removing from the cluster a failed node, the node'"'"'s failure being detected by the probing and using steps.

67. A computer storage medium having a configuration that represents data and instructions which will cause at least a portion of a computer system to perform method steps for managing resources in a cluster computing system, the method steps comprising the steps of claim 7.

13. A method for managing resources in a cluster, the method comprising the computer-implemented steps of:

remotely probing memory in a device in the cluster; and

using one of the following to determine whether the probed device has failed;

a counter value retrieved by the remote probing, or the unavailability of such a counter value;

wherein the probing step probes volatile memory in a remote system area network switch.

14. The method of claim 13, wherein the probing step and the using step are repeated at predetermined intervals to monitor the remote device.

15. The method of claim 13, wherein the probing step and the using step are performed in preparation for sending at least one of the following to the remote device:

data, or control information.

16. The method of claim 13, wherein the using step determines that the remote device has failed in response to determining that the counter value equals a previously retrieved counter value.

17. The method of claim 13, wherein the using step determines that the remote device has failed in response to the presence of a status value retrieved by the probing step.

18. The method of claim 13, further comprising the step of removing from the cluster a failed node, the node'"'"'s failure being detected by the probing and using steps.

68. A computer storage medium having a configuration that represents data and instructions which will cause at least a portion of a computer system to perform method steps for managing resources in a cluster computing system, the method steps comprising the steps of claim 13.

19. A method for managing resources in a cluster, the method comprising the computer-implemented steps of:

remotely probing memory in a device in the cluster; and

using one of the following to determine whether the probed device has failed;

a counter value retrieved by the remote probing, or the unavailability of such a counter value;

wherein the probing step probes memory in a remote cluster node and also probes memory in a remote system area network switch.

20. The method of claim 19, wherein the probing step and the using step are repeated at predetermined intervals to monitor the remote device.

21. The method of claim 19, wherein the probing step and the using step are performed in preparation for sending at least one of the following to the remote device:

data, or control information.

22. The method of claim 19, wherein the using step determines that the remote device has failed in response to determining that the counter value equals a previously retrieved counter value.

23. The method of claim 19, wherein the using step determines that the remote device has failed in response to the presence of a status value retrieved by the probing step.

24. The method of claim 19, further comprising the step of removing from the cluster a failed node, the node'"'"'s failure being detected by the probing and using steps.

69. A computer storage medium having a configuration that represents data and instructions which will cause at least a portion of a computer system to perform method steps for managing resources in a cluster computing system, the method steps comprising the steps of claim 19.

25. A method for managing resources in a cluster, the method comprising the computer-implemented steps of:

remotely probing memory in a device in the cluster; and

using one of the following to determine whether the probed device has failed;

a counter value retrieved by the remote probing, or the unavailability of such a counter value;

wherein the probing step probes nonvolatile memory in a shared storage device accessible by a remote cluster node.

26. The method of claim 25, wherein the probing step and the using step are repeated at predetermined intervals to monitor the remote device.

27. The method of claim 25, wherein the probing step and the using step are performed in preparation for sending at least one of the following to the remote device:

data, or control information.

28. The method of claim 25, wherein the using step determines that the remote device has failed in response to determining that the counter value equals a previously retrieved counter value.

29. The method of claim 25, wherein the using step determines that the remote device has failed in response to the presence of a status value retrieved by the probing step.

30. The method of claim 25, further comprising the step of removing from the cluster a failed node, the node'"'"'s failure being detected by the probing and using steps.

70. A computer storage medium having a configuration that represents data and instructions which will cause at least a portion of a computer system to perform method steps for managing resources in a cluster computing system, the method steps comprising the steps of claim 25.

31. A method for managing resources in a cluster, the method comprising the computer-implemented steps of:

remotely probing memory in a device in the cluster; and

using one of the following to determine whether the probed device has failed;

a counter value retrieved by the remote probing, or the unavailability of such a counter value;

wherein the using step is preceded by the step of determining the validity of the counter value.

32. The method of claim 31, wherein the probing step and the using step are repeated at predetermined intervals to monitor the remote device.

33. The method of claim 31, wherein the probing step and the using step are performed in preparation for sending at least one of the following to the remote device:

data, or control information.

34. The method of claim 31, wherein the using step determines that the remote device has failed in response to determining that the counter value equals a previously retrieved counter value.

35. The method of claim 31, wherein the using step determines that the remote device has failed in response to the presence of a status value retrieved by the probing step.

36. The method of claim 31, further comprising the step of removing from the cluster a failed node, the node'"'"'s failure being detected by the probing and using steps.

71. A computer storage medium having a configuration that represents data and instructions which will cause at least a portion of a computer system to perform method steps for managing resources in a cluster computing system, the method steps comprising the steps of claim 31.

37. A method for managing resources in a cluster, the method comprising the computer-implemented steps of:

remotely probing memory in a device in the cluster;

using one of the following to determine whether the probed device has failed;

a counter value retrieved by the remote probing, or the unavailability of such a counter value; and

removing from the cluster a failed node, the node'"'"'s failure being detected by the probing and using steps, wherein the removing step comprises writing to a predetermined emergency message location on a nonvolatile storage device that was and possibly still is accessible to the failed node.

38. The method of claim 37, wherein the probing step and the using step are repeated at predetermined intervals to monitor the remote device.

39. The method of claim 37, wherein the probing step and the using step are performed in preparation for sending at least one of the following to the remote device:

data, or control information.

40. The method of claim 37, wherein the using step determines that the remote device has failed in response to determining that the counter value equals a previously retrieved counter value.

41. The method of claim 37, wherein the using step determines that the remote device has failed in response to the presence of a status value retrieved by the probing step.

72. A computer storage medium having a configuration that represents data and instructions which will cause at least a portion of a computer system to perform method steps for managing resources in a cluster computing system, the method steps comprising the steps of claim 37.

42. A method for managing resources in a cluster, the method comprising the computer-implemented steps of:

remotely probing memory in a device in the cluster;

using one of the following to determine whether the probed device has failed;

a counter value retrieved by the remote probing, or the unavailability of such a counter value; and

removing from the cluster a failed node, the node'"'"'s failure being detected by the probing and using steps, wherein the removing step comprises returning to a global queue resources that were previously allocated to the failed node.

43. The method of claim 42, wherein the probing step and the using step are repeated at predetermined intervals to monitor the remote device.

44. The method of claim 42, wherein the probing step and the using step are performed in preparation for sending at least one of the following to the remote device:

data, or control information.

45. The method of claim 42, wherein the using step determines that the remote device has failed in response to determining that the counter value equals a previously retrieved counter value.

46. The method of claim 42, wherein the using step determines that the remote device has failed in response to the presence of a status value retrieved by the probing step.

73. A computer storage medium having a configuration that represents data and instructions which will cause at least a portion of a computer system to perform method steps for managing resources in a cluster computing system, the method steps comprising the steps of claim 42.

47. A computer system comprising:

at least two interconnected nodes capable of presenting a uniform system image such that an application program views the interconnected nodes as a single computing platform; and

a management means for managing computational resources for use by the nodes, wherein the management means comprises a means for detecting node failure by remotely probing memory.

48. A computer system comprising:

at least two interconnected nodes capable of presenting a uniform system image such that an application program views the interconnected nodes as a single computing platform; and

a management means for managing computational resources for use by the nodes, wherein the management means comprises a means for detecting interconnect failure by remotely probing memory.

49. A computer system comprising:

at least two interconnected nodes capable of presenting a uniform system image such that an application program views the interconnected nodes as a single computing platform; and

a management means for managing computational resources for use by the nodes, wherein the management means comprises a means for detecting system area network switch failure by remotely probing memory.

50. A computer system comprising:

at least two interconnected nodes capable of presenting a uniform system image such that an application program views the interconnected nodes as a single computing platform; and

a management means for managing computational resources for use by the nodes, wherein the management means comprises a means for one node to monitor a communication path to another node by remotely probing memory at regular time intervals.

51. The system of claim 50, wherein the probing intervals are staggered with respect to time intervals at which the probed memory is regularly updated during normal operation of the node being probed.

52. A computer system comprising:

at least two interconnected nodes capable of presenting a uniform system image such that an application program views the interconnected nodes as a single computing platform; and

a management means for managing computational resources for use by the nodes, wherein the management means comprises a means for a first node to monitor a communication path to a second node by remotely probing memory in preparation for sending a message from the first node to the second node.

53. A computer system comprising:

at least two interconnected nodes capable of presenting a uniform system image such that an application program views the interconnected nodes as a single computing platform; and

a management means for managing computational resources for use by the nodes, wherein the management means comprises a remote memory probing and evaluation means for a first node to distinguish between a first condition in which a second node and an interconnect connected to the second node are operating normally and a second condition in which the second node is restarting and the interconnect is operating normally.

54. A computer system comprising:

at least two interconnected nodes capable of presenting a uniform system image such that an application program views the interconnected nodes as a single computing platform; and

a management means for managing computational resources for use by the nodes, wherein the management means comprises a remote memory probing and evaluation means for a first node to distinguish between a first condition in which a second node and an interconnect connected to the second node are operating normally and a second condition in which the second node is operating normally after recently restarting and the interconnect is operating normally.

55. A computer system comprising:

at least two interconnected nodes capable of presenting a uniform system image such that an application program views the interconnected nodes as a single computing platform; and

a management means for managing computational resources for use by the nodes, wherein the management means comprises a remote memory probing and evaluation means for a first node to distinguish between a first condition in which a second node and an interconnect connected to the second node are operating normally and a second condition in which software running on the second node has failed and the interconnect is operating normally.

56. A computer system comprising:

at least two interconnected nodes capable of presenting a uniform system image such that an application program views the interconnected nodes as a single computing platform; and

a management means for managing computational resources for use by the nodes, wherein the management means comprises a remote memory probing and evaluation means for a first node to distinguish between a first condition in which a second node and an interconnect connected to the second node are operating normally and a second condition in which the second node has yielded control to a debugger and the interconnect is operating normally.

57. A computer system comprising:

at least two interconnected nodes capable of presenting a uniform system image such that an application program views the interconnected nodes as a single computing platform; and

a management means for managing computational resources for use by the nodes, wherein the management means comprises a remote memory probing and evaluation means for a first node to distinguish between a first condition in which a second node and an interconnect connected to the second node are operating normally and a second condition in which hardware within the second node has failed and the interconnect is operating normally.

58. A computer system comprising:

at least two interconnected nodes capable of presenting a uniform system image such that an application program views the interconnected nodes as a single computing platform; and

a management means for managing computational resources for use by the nodes, wherein the management means comprises a remote memory probing and evaluation means for a first node to distinguish between a first condition in which an interconnect connected to a second node is operating normally and a second condition in which the interconnect has failed.

59. A computer system comprising:

at least two interconnected nodes capable of presenting a uniform system image such that an application program views the interconnected nodes as a single computing platform; and

a management means for managing computational resources for use by the nodes, wherein the management means comprises a means for remotely probing memory to obtain a probe structure containing a counter.

60. The system of claim 59, wherein the management means comprises a means for remotely probing memory to obtain a probe structure further containing a validation field for validating the counter.

61. The system of claim 59, wherein at least one of the nodes is a special-purpose graphics node.

62. The system of claim 59, wherein at least one of the nodes is a special-purpose signal processing node.

63. A computer system comprising:

at least two interconnected nodes capable of presenting a uniform system image such that an application program views the interconnected nodes as a single computing platform; and

a management means for managing computational resources for use by the nodes, wherein the management means comprises a means for remotely probing memory to obtain a probe structure containing an epoch.

64. A computer system comprising:

at least two interconnected nodes capable of presenting a uniform system image such that an application program views the interconnected nodes as a single computing platform; and

a management means for managing computational resources for use by the nodes, wherein the management means comprises a means for remotely probing memory to obtain a probe structure containing a root pointer.

65. A computer system comprising:

at least two interconnected nodes capable of presenting a uniform system image such that an application program views the interconnected nodes as a single computing platform; and

a management means for managing computational resources for use by the nodes, wherein the management means comprises a means for remotely probing memory to obtain a probe structure containing a status summary.