Remote administration in a distributed system

• US 6,993,681 B2
• Filed: 04/15/2002
• Issued: 01/31/2006
• Est. Priority Date: 04/15/2002
• Status: Active Grant

• Current Assignee: General Electric Company
• Original Assignee: General Electric Company
• Inventors: Johnson, David N., Haynes, Leon E.
• Primary Examiner(s): BADERMAN, SCOTT T
• Application Number:
  US10/122,133
• Publication Number:
  US 20030196136A1
• Time in Patent Office: 1,387 Days
• Field of Search: 714/23, 714/4, 714/17, 714/43, 714/55, 714/46, 709/224
• US Class Current: 714/23
• CPC Class Codes: G06F 11/0715 in a system implementing multitasking multitasking per se G06F9/46
• CPC Class Codes: G06F 11/076 by exceeding a count or rate limit, e.g. word- or bit count limit
• CPC Class Codes: G06F 11/0793 Remedial or corrective actions recovery from an exception in an instruction pipeline G06F9/3861; by retry G06F11/1402; for recovering from a failure of a protocol instance or entity H04L69/40
• CPC Class Codes: G06F 11/324 Display of status information
• CPC Class Codes: G06F 11/3409 for performance assessment
• CPC Class Codes: G06F 11/3495 for systems
• CPC Class Codes: G06F 2201/81 Threshold
• CPC Class Codes: G06F 2201/86 Event-based monitoring

Abstract

A method and system for remote administration of clients performing critical processes are disclosed herein. A client management module is used to monitor a subset of processes performed by a client. When the client management module detects that a process of the subset has terminated or become inoperable, the client management module can restart the process, as well as notify a remote system manager. Additionally, the client management module is adapted to monitor the consumption of system resources by one or more processes. Using fuzzy logic to evaluate the system resource consumptions of one or more system resource types, the client management module can select an appropriate action response to improve the performance of the process and/or notify an administrator of a status of the process. The remote system manager is adapted to monitor a connection with the client. When a connection failure is detected, the system manager can diagnose a cause of the connection failure and take action accordingly, such as by resetting the client management module, resetting the client, or cycling power to the client using a power distribution unit. Additionally, the remote system manager provides various graphical user interfaces to a user or administrator to allow the administrator to assess the condition of the client and/or remotely control the operation of the one or more processes performed by the client.

Claims

1. In a distributed system having at least one system manager in bi-directional communication with at least one client performing a plurality of processes, a method for monitoring a subset of the plurality of processes comprising the steps of:

detecting a loss of supervisory control by the system manager over the processes at the client, including determining a status of a first process of the subset;

attempting by the client to restart the first process in the event the status indicates the first process is inoperative; and

resetting the client by cycling power thereto in the event the status of the first process is operative.

2. The method as in claim 1, further comprising the steps of:

determining the status of the first process subsequent to the attempted restarting of the first process;

attempting to restart the first process a second time when the status indicates the first process remains inoperative.

3. The method as in claim 1, wherein the steps of determining the status of the first process and attempting to restart the first process are repeated for up to a predetermined number of iterations until the status of the first process indicates the first process is operative.

4. The method as in claim 3, further comprising the step of resetting the client when the status of the first process continues to indicate the first process is inoperative after attempting to restart the first process for the predetermined number of iterations.

5. The method as in claim 1, further comprising the steps of:

determining a status of a second process of the subset; and

attempting to restart the second process when the status of the second process indicates the second process is inoperative.

6. In a distributed system having at least one system manager in bi-directional communication with at least one client performing a plurality of processes, a method for monitoring a subset of a plurality of processes at the client, the method comprising the steps of:

determining at least one system resource consumption by a first process of the subset;

performing a fuzzy logic evaluation of the at least one system resource consumption to determine a first action response to be performed by the client in response to the system resource consumption by the first process, the first action response including at least one action; and

performing, at the client, the at least one action of the first action response; and

wherein the step of performing the fuzzy logic evaluation includes the steps of;

determining at least one fuzzy logic truth value associated with the at least one system resource consumption;

evaluating at least one fuzzy logic rule using the at least one fuzzy logic truth value to determine a fuzzy logic solution;

defuzzifying the fuzzy logic solution to obtain an action response value; and

determining the first action response based on the action response value.

7. The method of claim 6, wherein the at least one system resource consumption by the first process is a central processing unit (CPU) usage, a memory usage, a rate-of-change in memory usage, a peak memory usage, a virtual memory size, a number of page faults, a rate-of-change in a number of page faults, a paged pool, a non-paged pool, a handle count, a thread count, a number of user objects, a graphical display interface object count, an input/output (I/O) read count, I/O read bytes, an I/O write count, or I/O write bytes.

8. The method of claim 6, wherein the at least one action is resetting the client, shutting down the first process, starting the first process, terminating the first process, displaying a query box for observation by an administrator/user, sending a page to an administrator'"'"'s pager, sending a facsimile to a fax machine, sending an email to an administrator, or sending a voice message to a mobile phone number.

9. The method of claim 6, wherein the first action response includes at least two iterations of an action.

10. The method of claim 6, wherein the step of determining the first action response includes selecting the action response from a plurality of action responses, each of the plurality of action responses having a different value range, and wherein a value range of the selected first action response includes the action response value.

11. The method of claim 6, further comprising the steps of:

determining at least one system resource consumption by a second process of the subset;

performing a fuzzy logic evaluation of the at least one system resource consumption by the second process to determine a second action response to be performed by the client in response to the system resource consumption by the second process, the second action response including at least one action; and

performing, at the client, the at least one action of the second action response.

12. In a distributed system having a system manager in bi-directional communication with at least one client, a method for monitoring a communications link between the system manager and a client, the method comprising the steps of:

detecting a failure in the communications link between the system manager and the client management module;

determining an operative status of a client management module at the client;

directing the client to restart the client management module when the operative status indicates the client management module is inoperative; and

resetting the client when the status indicates the client management module is operative.

13. The method of claim 12, wherein the step of determining the operative status is performed by attempting to contact the client management module.

14. The method of claim 13, further comprising the steps of:

determining a status of a connection of the system manager to the client over the communications link; and

restarting the client when the status indicates that the connection to the client is valid.

15. The method of claim 14, further comprising the steps of:

establishing a communication between the system manager and a power distribution unit adapted to provide power to the client;

and wherein the step of restarting the client includes the step of directing by the system manager the power distribution unit to cycle power to the client when the status indicates that the connection to the client is valid.

16. The method of claim 12, wherein the step of detecting the failure includes receiving a signal from a separate component in communication with the client, wherein the signal represents a communications failure between the separate component and the client.

17. In a distributed system having a system manager in bi-directional communication with at least one client having a client management module to monitor at least one process performed by the client, a graphical user interface for displaying at least one property of the client management module process at the system manager comprising:

a process status field having a user-selectable process entry for each of the at least one process, each process status field including;

a process identifier field representing an associated process;

a status field representing an operative status of the associated process; and

at least one resource consumption field representing a consumption by the associated process of an associated system resource;

a process operation object adapted to direct, when selected by a user, the client to restart a process associated with a process status field selected by the user when the associated process is inoperative and to stop the associated process when the associated process is operative; and

at least one client action object adapted to direct, when selected by a user, the client to perform an associated action.

18. The graphical user interface of claim 17, wherein the at least one client action object includes one of a group consisting of:

a shutdown object adapted to direct, when selected by a user, the client to shutdown the client;

a restart object adapted to direct, when selected by a user, the client to restart;

a cycle power object adapted to direct, when selected by a user, a power distribution unit to cycle power to the client; and

a ping object adapted to direct, when selected by a user, the system manager to determine a status of a connection of the system manager to the client.

19. The graphical user interface of claim 17, further comprising a client status field being adapted to display a status history of the client.

20. The graphical user interface of claim 17, further comprising:

a user field having a user-selectable user entry associated for each of at least one user of the client; and

a profile display object adapted to direct, when selected by a user, the client to display a profile of a user associated with a selected user entry.

21. In a distributed system having a system manager in bi-directional communication with at least one client having a client management module to monitor at least one process performed by the client, a graphical user interface for managing at least one property of the client management module at the system manager comprising:

a fuzzy logic field being adapted to receive input from a user, the input being representative of a desired change in at least one parameter of a fuzzy logic evaluation used by the client management module to manage one or more processes of the client; and

a modify object adapted to direct the system manager to transmit a signal representative of the desired change to the client management module for implementation at the client management module.

22. A system comprising:

a first client being adapted to perform a plurality of processes, the first client including a client management module adapted to;

determine at least one system resource consumption by a first process of plurality of processes;

perform a fuzzy logic evaluation of the at least one system resource consumption to determine a first action response to be performed by the first client in response to the system resource consumption by the first process, the first action response including at least one action;

perform the at least one action of the first action response; and

a system manager in communication with the first client and being adapted to direct the first client to modify at least one parameter of the fuzzy logic evaluation used by the first client.

23. The system of claim 22, wherein the system manager is further adapted to receive input from a user, the input being representative of a desired modification of the at least one parameter of the fuzzy logic evaluation performed by the first client.

24. The system of claim 22, wherein the system manager is further adapted to:

monitor a status of the client management module of the first client;

direct the first client to start the client management module when the status of the client management module indicates that the client management module is not activated at the first client; and

direct the first client to restart when the status of the client management module indicates that the client management module is inoperative.

25. The system of claim 24, wherein the system manager is further adapted to:

monitor a status of a connection between the first client and the system manager; and

direct the first client to restart when the status of the connection indicates that the connection is valid.

26. The system of claim 25, further including a power distribution unit being adapted to provide power to the first client.

27. The system of claim 26, wherein the system manager is further adapted to direct the power distribution unit to cycle power to the first client when the status of the connection indicates that the connection is invalid.

28. The system of claim 25, further comprising an external component in communication with the first client and the system manager and being adapted to provide a signal to the system manager when a failed connection between the external component and the first client is detected.

29. The system of claim 25, wherein the system manager is further adapted to provide for display a user a graphical user interface having an indicator of a connection status of the first client based at least in part on the notice from the first client.

30. The system of claim 25, wherein the system manager is further adapted to provide for display to a user a graphical user interface associated with the first client and having:

a process status field having a user-selectable process entry for each process of the subset of processes, each process status field including;

a process identifier field representing an associated process;

a status field representing an operative status of the associated process; and

at least one resource consumption field representing a consumption by the associated process of an associated system resource of the first client;

a process operation object adapted to direct, when selected by a user, the first client to restart a process associated with a selected process status field when the associated process is inoperative and to stop the associated process when the associated process is operative; and

at least one client action object adapted to direct, when selected by a user, the first client to perform an associated action.

31. The system of claim 30, wherein the at least one client action object includes one of a group consisting of:

a shutdown object adapted to direct, when selected by a user, the first client to shutdown;

a restart object adapted to direct, when selected by a user, the first client to restart;

a cycle power object adapted to direct, when selected by a user, a power distribution unit to cycle power to the first client; and

a ping object adapted to direct, when selected by a user, the system manager to determine a status of a connection of the system manager to the first client.

32. The system of claim 22, wherein the system manager is further adapted to provide for display to a user a graphical user interface associated with the first client and having:

a fuzzy logic field being adapted to receive input from a user, the input being representative of a desired change in at least one parameter of a fuzzy logic evaluation used by the client management module to manage one or more processes of the client; and

a process operation object adapted to direct the system manager to transmit a signal representative of the desired change to the client management module for implementation at the client management module.

33. The system of claim 22, further comprising a second client being adapted to perform a plurality of processes, the second client including a client management module adapted to:

determine at least one system resource consumption by a first process of plurality of processes performed by the second client;

perform a fuzzy logic evaluation of the at least one system resource consumption by the first process to determine a first action response to be performed by the second client in response to the system resource consumption by the first process, the second action response including at least one action; and

perform the at least one action of the second action response; and

wherein the system manager is further adapted to direct the second client to modify at least one parameter of the fuzzy logic evaluation used by the second client.

34. The system of claim 22, wherein a system resource consumed by the first process is a central processing unit (CPU) usage, a memory usage, a rate-of-change in memory usage, a peak memory usage, a virtual memory size, a number of page faults, a rate-of-change in a number of page faults, a paged pool, a non-paged pool, a handle count, a thread count, a number of user objects, a graphical display interface object count, an input/output (I/O) read count, I/O read bytes, an I/O write count, or I/O write bytes.

35. The system of claim 22, wherein the action is resetting the first client, shutting down the first process, starting the first process, terminating the first process, displaying a query box for observation by an administrator/user, sending a page to an administrator'"'"'s pager, sending a facsimile to a fax machine, sending an email to an administrator, or sending a voice message to a mobile phone number.

36. The system of claim 22, wherein the first client is further adapted to:

determine at least one system resource consumption by a second process of the subset;

perform a fuzzy logic evaluation of the at least one system resource consumption by the second process to determine a second action response to be performed by the first client in response to the system resource consumption by the second process, the second action response including at least one action; and

perform the at least one action of the second action response.

37. A computer readable medium being adapted to manipulate a processor to:

determine at least one system resource consumption by a first process of a subset of processes performed by a client;

perform a fuzzy logic evaluation of the at least one system resource consumption to determine a first action response to be performed by the client in response to the system resource consumption by the first process, the first action response including at least one action; and

perform the at least one action of the first action response; and

wherein the executable instructions being adapted to manipulate the processor to perform the fuzzy logic evaluation include executable instructions being adapted to;

determine at least one fuzzy logic truth value associated with the at least one system resource consumption;

evaluate at least one fuzzy logic rule using the at least one fuzzy logic truth value to determine a fuzzy logic solution;

defuzzify the fuzzy logic solution to obtain an action response value; and

determine the first action response based on the action response value.

38. The computer readable medium of claim 37, wherein the at least one system resource consumed by the first process is a central processing unit (CPU) usage, a memory usage, a rate-of-change in memory usage, a peak memory usage, a virtual memory size, a number of page faults, a rate-of-change in a number of page faults, a paged pool, a non-paged pool, a handle count, a thread count, a number of user objects, a graphical display interface object count, an input/output (I/O) read count, I/O read bytes, an I/O write count, or I/O write bytes.

39. The computer readable medium of claim 37, wherein the at least one action is resetting the client, shutting down the first process, starting the first process, terminating the first process, displaying a query box for observation by an administrator/user, sending a page to an administrator'"'"'s pager, sending a facsimile to a fax machine, sending an email to an administrator, or sending a voice message to a mobile phone number.

40. The computer readable medium of claim 37, wherein the first action response includes at least two iterations of an action.

41. The computer readable medium of claim 37, wherein the executable instructions being adapted to manipulate the processor to determine the first action response includes executable instructions being adapted to manipulate the processor to select the action response from a plurality of action responses, each of the plurality of action responses having a different value range, and wherein a value range of the selected first action response includes the action response value.