In-queue jobs information monitoring and filtering

• US 20030088425A1
• Filed: 11/06/2001
• Published: 05/08/2003
• Est. Priority Date: 11/06/2001
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Lam, Thanh V., Lauria, Giampaolo
• Granted Patent: US 7,177,823 B2
• Time in Patent Office: Days
• US Class Current: 705/1
• CPC Class Codes: G06Q 10/06 Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
• CPC Class Codes: G06Q 10/0631 Resource planning, allocation, distributing or scheduling for enterprises or organisations
• CPC Class Codes: G06Q 10/06314 Calendaring for a resource

Abstract

A remote monitor technique based on a client/server model is provided for a distributed computing system having a plurality of computing nodes. The remote monitor includes a server application for collecting jobs information on jobs in a queue of the system and a client application for presenting or employing the jobs information. The client application can display the jobs information in a single display window comprising multiple subwindows for various characteristics of the jobs information. Jobs information displayed by the client application can be periodically refreshed. The remote monitor enables a user to monitor in detail jobs of a state defined by the user. The remote monitor can also track usage of preselected filesystems and automatically trims filesystems if usage is higher than a defined threshold.

Claims

1. A method of monitoring jobs in a queue of a system, said method comprising:

collecting jobs information on jobs in a queue of a system, said jobs information comprising JobTracker information, said JobTracker information including jobs data on at least one job having a tracked job state and, for each job having said tracked job state, a time of entering the tracked job state; and

employing or displaying said jobs information on the jobs in the queue.

2. The method of claim 1, further comprising selecting said tracked job state.

3. The method of claim 2, wherein said selecting comprises selecting said tracked job state from a predefined list, said predefined list comprising at least some states of “

waiting”

, “

running”

, “

pending” and

“

held”

.

4. The method of claim 1, wherein said jobs information comprises multiple characteristics, said multiple characteristics including said JobTracker information, and at least one of jobs data, filesystem information, cluster information and core files information, said jobs data comprising for each job in a queue at least some of job id, job owner, job starting time, job state, job priority, job class, and name of node where the job is running, said filesystem information including filesystem usage, said cluster information including for each node of said system, a name of the node, status of a scheduler, number of in-queue jobs, number of active jobs, average load, idle time, type of node architecture and operating system running on the node, and said core files information including a list of core files.

5. The method of claim 4, wherein said collecting jobs information further comprises:

reading a set of nodes of said system;

querying nodes of said set of nodes on said jobs information; and

retrieving from said system results of said querying.

6. The method of claim 5, wherein said reading comprises:

retrieving from said system a full set of nodes of said system; and

removing down nodes from said full set of nodes, wherein said down nodes include nodes that fail to respond to operating system commands within a predefined period of time.

7. The method of claim 6, wherein said querying comprises at least some of:

querying nodes in said set of nodes on jobs information;

querying nodes in said set of nodes on JobTracker information;

querying nodes in said set of nodes on filesystem information;

querying nodes in said set of nodes on cluster information; and

querying nodes in said set of nodes on core files information.

8. The method of claim 7, wherein said querying nodes on JobTracker information comprises:

selecting said tracked job state from a predefined list of job states; and

producing a list of jobs in a queue of said system having said tracked job state.

9. The method of claim 8, wherein said producing a list of jobs comprises:

retrieving jobs information on at least one job in a queue and storing said jobs information into a list, wherein said jobs information includes a current timestamp and job state on said at least one job;

removing from said list jobs having a state different from said tracked job state; and

periodically updating said list by adding jobs that acquire said tracked job state after a most recent update of said list, and by removing from said list jobs that change job state after the most recent update of said list.

10. The method of claim 1, wherein said employing or displaying said jobs information comprises at least one of storing said jobs information, using said jobs information as an input of a computer program or a script, or presenting said jobs information in a single display window for viewing thereof.

11. A method of processing jobs information on jobs in a queue of a system, said method comprising:

collecting jobs information on jobs in a queue of a system, said jobs information comprising multiple characteristics; and

presenting said jobs information in one display window, wherein said one display window comprises multiple subwindows for displaying said jobs information, and at least some characteristics of said multiple characteristics are displayed in different subwindows of said multiple subwindows.

12. The method of claim 11, wherein said multiple characteristics comprise at least some of jobs data, JobTracker information, filesystem information, core files information and cluster information, wherein said jobs data comprises for each job in a queue at least some of job id, job owner, job starting time, job state, job priority, job class, and name of node where the job is running, said JobTracker information includes jobs data on jobs having a tracked job state and, for each job having said tracked job state, a time of entering the tracked job state, said filesystem information includes a filesystem usage, said core files information includes a list of core files, and said cluster information includes for each node of said system a name of the node, a status of a scheduler, number of in-queue jobs, number of active jobs, average load, idle time, type of node architecture and operating system running on the node.

13. The method of claim 12, further comprising selecting said tracked job state from a predefined list, said predefined list comprising at least some of states “

waiting”

, “

running”

, “

pending” and

“

held”

.

14. The method of claim 13, wherein said one display window further comprises a subwindow for selecting said tracked job state.

15. The method of claim 11, wherein said presenting further comprises periodically refreshing said jobs information.

16. The method of claim 15, wherein said one display window further comprises a subwindow for facilitating setting a cycle delay, said cycle delay controlling said periodically refreshing.

17. The method of claim 16, wherein said one display window further comprises at least one subwindow for facilitating stopping and resuming of said periodically refreshing said jobs information.

18. A method of monitoring jobs in a queue of a system, said method comprising:

providing a monitoring server application for collecting jobs information on jobs in a queue of a system, wherein said jobs information comprises multiple characteristics; and

providing a client monitor application, said client monitor application communicating with said monitoring server application and querying said monitoring server application for said jobs information on jobs in the queue of said system, said client monitor application facilitating displaying said jobs information on the jobs in the queue of said system.

19. The method of claim 18, wherein said client monitor application further comprises a user interface for displaying said multiple characteristics of jobs information in multiple subwindows of a single display window and for collecting user input for use in monitoring said jobs in said queue.

20. The method of claim 19, wherein said multiple characteristics comprise at least some of jobs data, JobTracker information, filesystem information, core files information and cluster information, wherein said jobs data comprises for each job in a queue at least some of job id, job owner, job starting time, job state, job priority, job class, and name of node where the job is running, said JobTracker information includes jobs data on jobs having a tracked job state and, for each job having said tracked job state, a time of entering the tracked job state, said filesystem information includes a filesystem usage, said core files information includes a list of core files, and said cluster information includes for each node of said system a name of the node, a status of a scheduler, number of in-queue jobs, number of active jobs, average load, idle time, type of node architecture, and operating system running on the node.

21. The method of claim 20, further comprising selecting said tracked job state from a predefined list, said predefined list comprising at least some states of “

waiting”

, “

running”

, “

pending” and

“

held”

.

22. The method of claim 20, wherein said collecting jobs information comprises:

reading a set of nodes of said system by said monitoring server application;

querying said system on said jobs information by said monitoring server application; and

retrieving from said system results of said querying by said monitoring server application.

23. The method of claim 22, wherein said reading comprises:

retrieving from said system a full set of nodes of said system; and

removing down nodes from said full set of nodes, wherein said down nodes include nodes failing to respond to operating system commands within a predefined period of time.

24. The method of claim 23, wherein said querying comprises:

sending a query on jobs information to said monitoring server application by said client monitor application; and

processing at least one of operating system commands and scripts on said set of nodes by said monitoring server application responsive to said query.

25. The method of claim 24, wherein said retrieving comprises:

reading an output of said at least one of operating system commands and scripts; and

sending said output to said monitoring client application by said monitoring server application.

26. The method of claim 25, wherein said displaying said jobs information comprises:

receiving said output by said monitoring client application; and

presenting said output in a single display window of said user interface.

27. A method of monitoring and trimming at least one filesystem of a system, said method comprising:

periodically collecting filesystem information from at least one filesystem of said system, wherein said filesystem information comprises a filesystem usage; and

automatically trimming said at least one filesystem of said system if said filesystem usage exceeds a predefined threshold.

28. The method of claim 27, wherein said at least one filesystem comprises at least one of a working directory and a temporary directory.

29. The method of claim 28, wherein said automatically trimming comprises automatically deleting a predefined number of files from said at least one filesystem having filesystem usage higher than said predefined threshold.

30. A monitor system for monitoring jobs in a queue of a system, said monitor system comprising:

means for collecting jobs information on jobs in a queue of a system, said jobs information comprising JobTracker information, said JobTracker information including jobs data on at least one job having a tracked job state and, for each job having said tracked job state, a time of entering the tracked job state; and

means for employing or displaying said jobs information on the jobs in the queue.

31. The monitor system of claim 30, further comprising means for selecting said tracked job state.

32. The monitor system of claim 31, wherein said means for selecting comprises means for selecting said tracked job state from a predefined list, said predefined list comprising at least some states of “

waiting”

, “

running”

, “

pending” and

“

held”

.

33. The monitor system of claim 30, wherein said jobs information comprises multiple characteristics, said multiple characteristics including said JobTracker information, and at least one of jobs data, filesystem information, cluster information and core files information, said jobs data comprising for each job in a queue at least some of job id, job owner, job starting time, job state, job priority, job class, and name of node where the job is running, said filesystem information including filesystem usage, said cluster information including for each node of said system, a name of the node, status of a scheduler, number of in-queue jobs, number of active jobs, average load, idle time, type of node architecture and operating system running on the node, and said core files information including a list of core files.

34. The monitor system of claim 33, wherein said means for collecting jobs information further comprises:

means for reading a set of nodes of said system;

means for querying nodes of said set of nodes on said jobs information; and

means for retrieving from said system results of said means for querying.

35. The monitor system of claim 34, wherein said means for reading comprises:

means for retrieving from said system a full set of nodes of said system; and

means for removing down nodes from said full set of nodes, wherein said down nodes include nodes that fail to respond to operating system commands within a predefined period of time.

36. The monitor system of claim 35, wherein said means for querying comprises at least some of:

means for querying nodes in said set of nodes on jobs information;

means for querying nodes in said set of nodes on JobTracker information;

means for querying nodes in said set of nodes on filesystem information;

means for querying nodes in said set of nodes on cluster information; and

means for querying nodes in said set of nodes on core files information.

37. The monitor system of claim 36, wherein said means for querying nodes on JobTracker information comprises:

means for selecting said tracked job state from a predefined list of job states; and

means for producing a list of jobs in a queue of said system having said tracked job state.

38. The monitor system of claim 37, wherein said means for producing a list of jobs comprises:

means for retrieving jobs information on at least one job in a queue and means for storing said jobs information into a list, wherein said jobs information includes a current timestamp and job state on said at least one job;

means for removing from said list jobs having a state different from said tracked job state; and

means for periodically updating said list by adding jobs that acquire said tracked job state after a most recent update of said list, and by removing from said list jobs that change job state after the most recent update of said list.

39. The monitor system of claim 30, wherein said means for employing or displaying said jobs information comprises at least one of means for storing said jobs information, means for using said jobs information as an input of a computer program or a script, or means for presenting said jobs information in a single display window for viewing thereof.

40. A monitor system for monitoring jobs in a queue of a system, said monitor system comprising:

a computing unit adapted to collect jobs information on jobs in a queue of a system, said jobs information comprising JobTracker information, said JobTracker information including jobs data on at least one job having a tracked job state and, for each job having said tracked job state, a time of entering the tracked job state; and

wherein said at least one computing unit is further adapted to employ or display said jobs information on the jobs in the queue.

41. A monitor system for processing jobs information on jobs in a queue of a system, said monitor system comprising:

means for collecting jobs information on jobs in a queue of a system, said jobs information comprising multiple characteristics; and

means for presenting said jobs information in one display window, wherein said one display window comprises multiple subwindows for displaying said jobs information, and at least some characteristics of said multiple characteristics are displayed in different subwindows of said multiple subwindows.

42. The monitor system of claim 41, wherein said multiple characteristics comprise at least some of jobs data, JobTracker information, filesystem information, core files information and cluster information, wherein said jobs data comprises for each job in a queue at least some of job id, job owner, job starting time, job state, job priority, job class, and name of node where the job is running, said JobTracker information includes jobs data on jobs having a tracked job state and, for each job having said tracked job state, a time of entering the tracked job state, said filesystem information includes a filesystem usage, said core files information includes a list of core files, and said cluster information includes for each node of said system a name of the node, a status of a scheduler, number of in-queue jobs, number of active jobs, average load, idle time, type of node architecture and operating system running on the node.

43. The monitor system of claim 42, further comprising means for selecting said tracked job state from a predefined list, said predefined list comprising at least some of states “

waiting”

, “

running”

, “

pending” and

“

held”

.

44. The monitor system of claim 43, wherein said one display window further comprises a subwindow comprising means for selecting said tracked job state.

45. The monitor system of claim 41, wherein said means for presenting further comprises means for periodically refreshing said jobs information.

46. The monitor system of claim 45, wherein said one display window further comprises a subwindow comprising means for facilitating setting a cycle delay, said cycle delay controlling said periodically refreshing.

47. The monitor system of claim 46, wherein said one display window further comprises at least one subwindow comprising means for facilitating stopping and resuming of said periodically refreshing said jobs information.

48. A monitor system for processing jobs information on jobs in a queue of a system, said monitor system comprising:

a computing unit adapted to collect jobs information on jobs in a queue of a system, said jobs information comprising multiple characteristics; and

wherein said at least one computing unit is further adapted to present said jobs information in one display window, said one display window comprising multiple subwindows for displaying said jobs information, at least some characteristics of said multiple characteristics being displayed in different subwindows of said multiple subwindows.

49. A monitor system for monitoring jobs in a queue of a system, said monitor system comprising:

means for providing a monitoring server application for collecting jobs information on jobs in a queue of a system, wherein said jobs information comprises multiple characteristics; and

means for providing a client monitor application for communicating with said monitoring server application and querying said monitoring server application for said jobs information on jobs in the queue of said system, said client monitor application facilitating displaying said jobs information on the jobs in the queue of said system.

50. The monitor system of claim 49, wherein said A client monitor application further comprises a user interface for displaying said multiple characteristics of jobs information in multiple subwindows of a single display window and for collecting user input for use in monitoring said jobs in said queue.

51. The monitor system of claim 49, wherein said multiple characteristics comprise at least some of jobs data, JobTracker information, filesystem information, core files information and cluster information, wherein said jobs data comprises for each job in a queue at least some of job id, job owner, job starting time, job state, job priority, job class, and name of node where the job is running, said JobTracker information includes jobs data on jobs having a tracked job state and, for each job having said tracked job state, a time of entering the tracked job state, said filesystem information includes a filesystem usage, said core files information includes a list of core files, and said cluster information includes for each node of said system a name of the node, a status of a scheduler, number of in-queue jobs, number of active jobs, average load, idle time, type of node architecture, and operating system running on the node.

52. The monitor system of claim 51, further comprising means for selecting said tracked job state from a predefined list, said predefined list comprising at least some states of “

waiting”

, “

running”

, “

pending” and

“

held”

.

53. The monitor system of claim 51, wherein said collecting jobs information by said monitoring server application comprises:

reading a set of nodes of said system;

querying said system on said jobs information; and

retrieving from said system results of said querying.

54. The monitor system of claim 53, wherein said reading by said monitoring server application comprises:

retrieving from said system a full set of nodes of said system; and

removing down nodes from said full set of nodes, wherein said down nodes include nodes failing to respond to operating system commands within a predefined period of time.

55. The monitor system of claim 54, wherein said querying comprises:

sending a query on jobs information to said monitoring server application by said client monitor application; and

processing at least one of operating system commands and scripts on said set of nodes by said monitoring server application responsive to said query.

56. The monitor system of claim 55, wherein said retrieving comprises:

reading an output of said at least one of operating system commands and scripts; and

sending said output to said monitoring client application by said monitoring server application.

57. The monitor system of claim 56, wherein said displaying said jobs information comprises:

receiving said output by said monitoring client application; and

presenting said output in a single display window of said user interface.

58. A monitor system for monitoring jobs in a queue of a system, said monitor system comprising:

a monitoring server application for collecting jobs information on jobs in a queue of a system, wherein said jobs information comprises multiple characteristics; and

a client monitor application communicating with said monitoring server application and querying said monitoring server application for said jobs information on jobs in the queue of said system, said client monitor application facilitating displaying said jobs information on the jobs in the queue of the system.

59. A monitor system for monitoring and trimming at least one filesystem of a system, said monitor system comprising:

means for periodically collecting filesystem information from at least one filesystem of said system, wherein said filesystem information comprises a filesystem usage; and

means for automatically trimming said at least one filesystem of said system should said filesystem usage exceeds a predefined threshold.

60. The monitor system of claim 59, wherein said at least one filesystem comprises at least one of a working directory and a temporary directory.

61. The monitor system of claim 60, wherein said means for automatically trimming comprises means for automatically deleting a predefined number of files from said at least one filesystem having filesystem usage higher than said predefined threshold.

62. A monitor system for monitoring and trimming at least one filesystem of a system, said monitor system comprising:

at least one computing unit adapted to periodically collect filesystem information from at least one filesystem of said system, wherein said filesystem information comprises a filesystem usage; and

wherein said at least one computing unit is further adapted to automatically trim said at least one filesystem of said system should said filesystem usage exceed a predefined threshold.

63. At least one program storage device readable by a machine, tangibly embodying at least one program of instructions executable by the machine to perform a method of monitoring jobs in a queue of a system, comprising:

collecting jobs information on jobs in a queue of a system, said jobs information comprising JobTracker information, said JobTracker information including jobs data on at least one job having a tracked job state and, for each job having said tracked job state, a time of entering the tracked job state; and

employing or displaying said jobs information on the jobs in the queue.

64. At least one program storage device readable by a machine, tangibly embodying at least one program of instructions executable by the machine to perform a method of processing jobs information on jobs in a queue of a system, comprising:

collecting jobs information on jobs in a queue of a system, said jobs information comprising multiple characteristics; and

presenting said jobs information in one display window, wherein said one display window comprises multiple subwindows for displaying said jobs information, and at least some characteristics of said multiple characteristics are displayed in different subwindows of said multiple subwindows.

65. At least one program storage device readable by a machine, tangibly embodying at least one program of instructions executable by the machine to perform a method of monitoring jobs in a queue of a system, comprising:

providing a monitoring server application for collecting jobs information on jobs in a queue of a system, wherein said jobs information comprises multiple characteristics; and

providing a client monitor application, said client monitor application communicating with said monitoring server application and querying said monitoring server application for said jobs information on jobs in the queue of said system, said client monitor application facilitating displaying said jobs information on the jobs in the queue of said system.

66. At least one program storage device readable by a machine, tangibly embodying at least one program of instructions executable by the machine to perform a method of monitoring and trimming at least one filesystem of a system, comprising:

periodically collecting filesystem information from at least one filesystem of said system, wherein said filesystem information comprises a filesystem usage; and

automatically trimming said at least one filesystem of said system should said filesystem usage exceed a predefined threshold.