SYSTEM AND METHOD OF FAULT DETECTION, DIAGNOSIS AND PREVENTION FOR COMPLEX COMPUTING SYSTEMS

• US 20090193298A1
• Filed: 01/30/2008
• Published: 07/30/2009
• Est. Priority Date: 01/30/2008
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Mukherjee, Maharaj
• Granted Patent: US 8,949,671 B2
• Time in Patent Office: Days
• US Class Current: 714/38.1
• CPC Class Codes: G06F 11/0718 in an object-oriented system
• CPC Class Codes: G06F 11/0766 Error or fault reporting or storing

Abstract

A method is provided for diagnosing failures in an object-oriented software system. The method comprises collecting runtime diagnostic information; maintaining a record of the diagnostic information in a storage buffer; and dynamically updating the record of the diagnostic information to include a group of the diagnostic information collected over a most recent occurrence of a predetermined interval. The diagnostic information includes at least one set of call stack information for at least one currently running application and at least one set of other information. Each of the at least one set of other information is selected from a set of memory access information, a set of data access information, and a set of paging information for each currently executing process.

Claims

1. A method for diagnosing failures in an object-oriented software system, the method comprising:

collecting runtime diagnostic information, the diagnostic information including at least one set of call stack information for at least one currently running application and at least one set of other information, each of the at least one other set of information being selected from a set of memory access information, a set of data access information, and a set of paging information for each currently executing process;

maintaining a record of the diagnostic information in a storage buffer; and

dynamically updating the record of the diagnostic information to include a group of the diagnostic information collected over a most recent occurrence of a predetermined interval.

2. The method of claim 1, further comprising monitoring the software system to detect occurrences of failures related to runtime interactions between a set of objects in the software system, and evaluating the record of the diagnostic information to attempt to localize a cause of each occurrence of a failure that is detected.

3. The method of claim 1, further comprising monitoring a set of resources being utilized by the software system, and recording a snapshot of the record of the diagnostic information whenever any resource of the set of resources is being utilized beyond a specified threshold for that resource.

4. The method of claim 2, further comprising monitoring a set of resources being utilized by the software system, and recording a snapshot of the record of the diagnostic information whenever any of the set of resources is being utilized beyond a specified threshold for that resource.

5. The method of claim 4, wherein the method is performed by one or more separate processes executing within an address space of the software system, one or more threads of a process executing within the software system, one or more daemons executing within the software system, one or more services executing outside the address space of the software system, or combinations thereof.

6. The method of claim 1, wherein the diagnostic information is generated by one or more processes executing within the software system.

7. The method of claim 1, wherein the diagnostic information is generated by one or more threads of one or more processes executing within the software system.

8. The method of claim 1, wherein the software system is communicatively coupled to a data repository, and wherein the storage buffer is maintained in the data repository.

9. The method of claim 1, wherein the record of the diagnostic information is maintained in a log trace file.

10. The method of claim 1, wherein the predetermined interval is specified according to a maximum size for the storage buffer, a maximum number of call stack changes, or a maximum period of time.

11. The method of claim 2, further comprising allowing an analyst to access the record of the diagnostic information to attempt identify and categorize the cause of each occurrence of a failure that is detected.

12. The method of claim 11, further comprising sorting and extracting information from the record of diagnostic information that is relevant to each occurrence of a failure that is detected.

13. The method of claim 3, wherein monitoring the set of resources comprises monitoring a set of resource parameters selected from CPU utilization for one or more processors, memory utilization of logical and physical memory, page file usage, disk I/O utilization, a number of processes or threads concurrently being executed, length of a data access wait list, network throughput, and combinations thereof.

14. The method of claim 3, wherein the software system is communicatively coupled to a data repository, and wherein each snapshot of the record of diagnostic information that is recorded is maintained in a record of snapshots in the data repository.

15. The method of claim 14, further comprising utilizing the record of snapshots to predict occurrences of failures related to runtime interactions between a set of objects in the software system.

16. The method of claim 15, wherein utilizing the record of snapshots to predict occurrences of failures comprises performing regular regression testing.

17. The method of claim 15, wherein utilizing the record of snapshots to predict occurrences of failures comprises creating a failure model of data clusters for utilization conditions of the set of resources.

18. The method of claim 1, wherein the software system is an application selected from operating system applications, database management applications, server-side software applications, web-base applications, and client-side software applications.

19. The method of claim 1, wherein the software system is executing on a single processor, several processors in close proximity, or distributed across a network.

20. A system for diagnosing failures in an object-oriented software system, the system comprising:

a first software module configured to collect runtime diagnostic information, the diagnostic information including at least one set of call stack information for at least one currently running application in the software system, and at least one set of other information, each of the at least one set of other information being selected from a set of memory access information, a set of data access information, and a set of paging information for each currently executing process; and

a storage buffer configured to maintain a record of the diagnostic information, the storage buffer being configured to automatically receive the diagnostic information collected by the first software module and dynamically update the record to include a group of the diagnostic information collected over a most recent occurrence of a predetermined interval.

21. The diagnostic system of claim 20, further comprising a second software module configured to be notified by the software system of each occurrence of a failure related to runtime interactions between a set of objects in the software system, the second software module being configured to access the storage buffer to evaluate the record of the diagnostic information to attempt to localize a cause of each occurrence of a failure for which the second software module receives notification.

22. The diagnostic system of claim 21, further comprising a third software module configured to monitor a set of resources being utilized by the software system, the third software module being further configured to record a snapshot of the record of the diagnostic information whenever any resource of the set of resources is being utilized beyond a specified threshold for that resource.

23. The diagnostic system of claim 22, wherein the first, second, and third software modules are implemented within one or more libraries of functions, one or more plug-in modules, one or more dynamic link-libraries, or combinations thereof.

24. A computer-usable medium having computer readable instructions stored thereon for execution by a processor to perform a method for diagnosing failures in an object-oriented software system, the method comprising:

collecting runtime diagnostic information, the diagnostic information including at least one set of call stack information for at least one currently running application and at least one set of other information, each of the at least one set of other information being selected from a set of memory access information, a set of data access information, and a set of paging information for each currently executing process;

maintaining a record of the diagnostic information in a storage buffer; and

dynamically updating the record of the diagnostic information to include a group of the diagnostic information collected over a most recent occurrence of a predetermined interval.

25. The computer-usable medium of claim 24, wherein the method further comprises monitoring the software system to detect occurrences of failures related to runtime interactions between a set of objects in the software system, and evaluating the record of the diagnostic information to attempt to localize a cause of each occurrence of a failure that is detected.

26. The computer-usable medium of claim 24, wherein the method further comprises monitoring a set of resources being utilized by the software system, and recording a snapshot of the record of the diagnostic information whenever any resource of the set of resources is being utilized beyond a specified threshold for that resource.

27. A data processing system comprising:

a central processing unit;

a random access memory for storing data and programs for execution by the central processing unit;

a first storage level comprising a nonvolatile storage device; and

computer readable instructions stored in the random access memory for execution by central processing unit to perform a method for diagnosing failures in an object-oriented software system, the method comprising;

collecting runtime diagnostic information, the diagnostic information including at least one set of call stack information for at least one currently running application and at least one set of other information, each of the at least one set of other information being selected from a set of memory access information, a set of data access information, and a set of paging information for each currently executing process;

maintaining a record of the diagnostic information in a storage buffer; and

dynamically updating the record of the diagnostic information to include a group of the diagnostic information collected over a most recent occurrence of a predetermined interval.

28. The data processing system of claim 27, wherein the method further comprises monitoring the software system to detect occurrences of failures related to runtime interactions between a set of objects in the software system, and evaluating the record of the diagnostic information to attempt to localize a cause of each occurrence of a failure that is detected.

29. The data processing system of claim 27, wherein the method further comprises monitoring a set of resources being utilized by the software system, and recording a snapshot of the record of the diagnostic information whenever any resource of the set of resources is being utilized beyond a specified threshold for that resource.