Diagnosing an incident on a computer system using a diagnostics analyzer database

• US 8,327,189 B1
• Filed: 12/22/2009
• Issued: 12/04/2012
• Est. Priority Date: 12/22/2009
• Status: Active Grant

• Current Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Original Assignee: EMC Corporation (Dell Technologies Inc.)
• Inventors: Glade, Bradford B., Gurevich, Naum
• Primary Examiner(s): TRUONG, LOAN
• Application Number:
  US12/644,921
• Time in Patent Office: 1,078 Days
• Field of Search: 714/26, 714/37, 714/38.11, 714/39, 714/45
• US Class Current: 714/26
• CPC Class Codes: G06F 11/079 Root cause analysis, i.e. error or fault diagnosis in a hardware test environment G06F11/22; in a software test environment G06F11/36

Abstract

A technique diagnoses an incident on a computer system. The technique involves electronically receiving a bundle of diagnostic information from the computer system and storing the diagnostic information in memory after the incident on the computer system has occurred. The technique further involves electronically applying a set of diagnostics analyzers (e.g., diagnostics analyzers for discovering known incident signatures and invariant violations) from a diagnostics analyzer database to the diagnostic information after the bundle of diagnostic information is electronically received from the computer system and stored in the memory. The electronic application of the set of diagnostics analyzers from the diagnostics analyzers database to the diagnostic information results in a set of analyzer results. The technique further involves electronically generating a report which identifies a reason for the incident on the computer system based on the set of analyzer results.

Claims

1. A method of diagnosing an incident on a computer system, the method comprising:

after the incident on the computer system has occurred, electronically receiving a bundle of diagnostic information from the computer system and storing the diagnostic information in a memory;

after electronically receiving the bundle of diagnostic information from the computer system and storing the diagnostic information in the memory, electronically applying a set of diagnostics analyzers from a diagnostics analyzer database to the diagnostic information using processing circuitry which is coupled to the diagnostics analyzer database, the electronic application of the set of diagnostics analyzers from the diagnostics analyzer database to the diagnostic information resulting in a set of analyzer results; and

electronically generating a report which identifies a reason for the incident on the computer system based on the set of analyzer results;

wherein the diagnostics analyzer database stores fault signature detector entries, each fault signature detector entry including (i) a fault signature, (ii) a known fault associated with that fault signature, and (iii) a recommendation to remedy that known fault;

wherein electronically applying the set of diagnostics analyzers from the diagnostics analyzer database to the diagnostic information includes;

scanning the diagnostic information for the fault signature of each fault signature detector entry stored in the diagnostics analyzer database, and

providing a fault indication signal identifying whether the fault signature of any fault signature detector entry was discovered in the diagnostic information during scanning of the diagnostic information for the fault signature of each fault signature detector entry;

wherein the diagnostics analyzer database further stores invariant violation detector entries, each invariant violation detector entry including a known invariant condition which is needed for proper operation of the computer system;

wherein electronically applying the set of diagnostics analyzers from the diagnostics analyzer database to the diagnostic information further includes;

scanning the diagnostic information for the known invariant condition of each invariant violation detector entry stored in the diagnostics analyzer database, and

providing an invariant indication signal identifying whether the known invariant condition of any invariant violation detector entry was violated based on scanning the diagnostic information for the known invariant condition of each invariant violation detector entry; and

wherein electronically applying the set of diagnostics analyzers from the diagnostics analyzer database to the diagnostic information is complete only after (i) the fault indication signal has been provided and (ii) the respective invariant indication signal has been provided, the fault indication signal and the invariant indication signal forming at least a portion of the set of analyzer results.

2. A method as in claim 1 wherein electronically generating the report which identifies the reason for the incident on the computer system based on the set of analyzer results includes:

providing a confidence level signal indicating a level of confidence for the reason for the incident among other possible reasons for the incident, the level of confidence being based on a weighted analysis of the set of analyzer results.

3. A method as in claim 1 wherein electronically generating the report which identifies the reason for the incident on the computer system based on the set of analyzer results includes:

outputting, as the report, a list containing (i) known root cause information as the reason for the incident, (ii) a known workaround which temporarily prevents reoccurrence of the incident, and (iii) known resolution information which permanently prevents reoccurrence of the incident.

4. A method as in claim 1 wherein electronically receiving and storing the diagnostic information includes:

obtaining, as the bundle of diagnostic information, an electronic transmission from the computer system, the electronic transmission containing (i) a set of trace files produced by a set of applications which was running on the computer system when the incident on the computer system occurred, (ii) a set of process images defining memory contents of a set of processes running the set of applications on the computer system, and (iii) a set of state snapshots identifying a set of current operating states of the computer system when the incident on the computer system occurred, and

storing the set of trace files, the set of process images, and the set of state snapshots in separate locations within the memory for individual searching using the diagnostics analyzers.

5. A method as in claim 4 wherein the computer system includes multiple processing nodes connected together on a computer network;

and

wherein obtaining the electronic transmission from the computer system includes acquiring, from each processing node, (i) at least one trace file from that processing node, (ii) at least one process image from that processing node, and (iii) at least one state snapshot from that processing node.

6. A method as in claim 5 wherein the multiple processing nodes

of the computer system are constructed and arranged to operate as a data storage cluster;

and

wherein the method further comprises;

implementing a remedy on the data storage cluster based on the reason identified on the electronically generated report to cure operation of the data storage cluster.

7. A method as in claim 6 wherein the data storage cluster resides at a field location, and the processing circuitry and the diagnostics analyzer database reside at an analyzer location which is separate from the field location;

and

wherein the method further comprises;

updating the diagnostics analyzer database residing at the analyzer location with new fault signature detector entries as new faults are discovered and new remedies are developed over time.

8. A method as in claim 7, further comprising:

deleting outdated fault signature detector entries from the diagnostics analyzer database residing at the analyzer location as old faults and old remedies of the outdated fault signature entries become obsolete over time.

9. An analyzer apparatus to diagnose an incident on a computer system, the analyzer comprising:

memory;

a diagnostics analyzer database; and

processing circuitry coupled to the memory and the diagnostics analyzer database, the processing circuitry being constructed and arranged to;

electronically receive a bundle of diagnostic information from the computer system and store the diagnostic information in the memory after the incident on the computer system has occurred,

electronically apply a set of diagnostics analyzers from the diagnostics analyzer database to the diagnostic information after the bundle of diagnostic information is electronically received from the computer system and stored in the memory, the electronic application of the set of diagnostics analyzers from the diagnostics analyzer database to the diagnostic information resulting in a set of analyzer results, and

electronically generate a report which identifies a reason for the incident on the computer system based on the set of analyzer results;

wherein the diagnostics analyzer database stores fault signature detector entries, each fault signature detector entry including (i) a fault signature, (ii) a known fault associated with that fault signature, and (iii) a recommendation to remedy that known fault;

wherein the processing circuitry, when electronically applying the set of diagnostics analyzers from the diagnostics analyzer database to the diagnostic information, is constructed and arranged to;

scan the diagnostic information for the fault signature of each fault signature detector entry stored in the diagnostics analyzer database, and

provide a fault indication signal identifying whether the fault signature of any fault signature detector entry was discovered in the diagnostic information during scanning of the diagnostic information for the fault signature;

wherein the diagnostics analyzer database further stores invariant violation detector entries, each invariant violation detector entry including a known invariant condition which is needed for proper operation of the computer system;

wherein the processing circuitry, when electronically applying the set of diagnostics analyzers from the diagnostics analyzer database to the diagnostic information, is further constructed and arranged to;

scan the diagnostic information for the known invariant condition of each invariant violation detector entry stored in the diagnostics analyzer database, and

provide an invariant indication signal identifying whether the known invariant condition of any invariant violation detector entry was satisfied based on scanning the diagnostic information for the known invariant condition; and

wherein electronically applying the set of diagnostics analyzers from the diagnostics analyzer database to the diagnostic information is complete only after (i) the fault indication signal has been provided and (ii) the respective invariant indication signal has been provided, the fault indication signal and the invariant indication signal forming at least a portion of the set of analyzer results.

10. An analyzer apparatus as in claim 9 wherein the processing circuitry, when electronically generating the report which identifies the reason for the incident on the computer system based on the set of analyzer results, is constructed and arranged to:

provide a confidence level signal indicating a level of confidence for the reason for the incident among other possible reasons for the incident, the level of confidence being based on a weighted analysis of the set of analyzer results.

11. An analyzer apparatus as in claim 9 wherein the processing circuitry, when electronically generating the report which identifies the reason for the incident on the computer system based on the set of analyzer results, is constructed and arranged to:

output, as the report, a list containing (i) known root cause information as the reason for the incident, (ii) a known workaround which temporarily prevents reoccurrence of the incident, and (iii) known resolution information which permanently prevents reoccurrence of the incident.

12. An analyzer apparatus as in claim 9 wherein the processing circuitry, when electronically receiving and storing the bundle of diagnostic information, is constructed and arranged to:

obtain, as the bundle of diagnostic information, an electronic transmission from the computer system, the electronic transmission containing (i) a set of trace files produced by a set of applications which was running on the computer system when the incident on the computer system occurred, (ii) a set of process images defining memory contents of a set of processes running the set of applications on the computer system, and (iii) a set of state snapshots identifying a set of current operating states of the computer system when the incident on the computer system occurred, and

store the set of trace files, the set of process images, and the set of state snapshots in separate locations within the memory for individual searching using the diagnostics analyzers.

13. An analyzer apparatus as in claim 12 wherein the computer system includes multiple processing nodes connected together on a computer network;

and

wherein the processing circuitry, when obtaining the electronic transmission from the computer system, is constructed and arranged to acquire, from each processing node of the computer system, (i) at least one trace file from that processing node, (ii) at least one process image from that processing node, and (iii) at least one state snapshot from that processing node.

14. An analyzer apparatus as in claim 13, further comprising:

a user interface coupled to the processing circuitry;

the processing circuitry being constructed and arranged to respond to user commands, which are received through the user interface, by (i) updating the diagnostics analyzer database with new fault signature detector entries as new faults are discovered and new remedies are developed over time, and (ii) deleting outdated fault signature detector entries from the diagnostics analyzer database as old faults and old remedies of the outdated fault signature entries become obsolete over time.

15. A computer program product which includes a non-transitory computer readable medium storing a set of instructions to diagnose an incident on a computer system, the set of instructions causing computing circuitry to perform a method, comprising:

after the incident on the computer system has occurred, electronically receiving a bundle of diagnostic information from the computer system and storing the diagnostic information in a memory;

after electronically receiving the bundle of diagnostic information from the computer system and storing the diagnostic information in the memory, electronically applying a set of diagnostics analyzers from a diagnostics analyzer database to the diagnostic information using processing circuitry which is coupled to the diagnostics analyzer database, the electronic application of the set of diagnostics analyzers from the diagnostics analyzer database to the diagnostic information resulting in a set of analyzer results; and

electronically generating a report which identifies a reason for the incident on the computer system based on the set of analyzer results;

wherein the diagnostics analyzer database stores fault signature detector entries, each fault signature detector entry including (i) a fault signature, (ii) a known fault associated with that fault signature, and (iii) a recommendation to remedy that known fault;

wherein electronically applying the set of diagnostics analyzers from the diagnostics analyzer database to the diagnostic information includes;

scanning the diagnostic information for the fault signature of each fault signature detector entry stored in the diagnostics analyzer database, and

providing a fault indication signal identifying whether the fault signature of any fault signature detector entry was discovered in the diagnostic information during scanning of the diagnostic information for the fault signature of each fault signature detector entry;

wherein the diagnostics analyzer database further stores invariant violation detector entries, each invariant violation detector entry including a known invariant condition which is needed for proper operation of the computer system;

wherein electronically applying the set of diagnostics analyzers from the diagnostics analyzer database to the diagnostic information further includes;

scanning the diagnostic information for the known invariant condition of each invariant violation detector entry stored in the diagnostics analyzer database, and

providing an invariant indication signal identifying whether the known invariant condition of any invariant violation detector entry was violated based on scanning the diagnostic information for the known invariant condition of each invariant violation detector entry; and

wherein electronically applying the set of diagnostics analyzers from the diagnostics analyzer database to the diagnostic information is complete only after (i) the fault indication signal has been provided and (ii) the respective invariant indication signal has been provided, the fault indication signal and the invariant indication signal forming at least a portion of the set of analyzer results.

16. A computer program product as in claim 15 wherein electronically generating the report which identifies the reason for the incident on the computer system based on the set of analyzer results includes:

providing a confidence level signal indicating a level of confidence for the reason for the incident among other possible reasons for the incident, the level of confidence being based on a weighted analysis of the set of analyzer results.

17. A computer program product as in claim 15 wherein electronically generating the report which identifies the reason for the incident on the computer system based on the set of analyzer results includes:

outputting, as the report, a list containing (i) known root cause information as the reason for the incident, (ii) a known workaround which temporarily prevents reoccurrence of the incident, and (iii) known resolution information which permanently prevents reoccurrence of the incident.

18. A computer program product as in claim 15 wherein electronically receiving and storing the diagnostic information includes:

obtaining, as the bundle of diagnostic information, an electronic transmission from the computer system, the electronic transmission containing (i) a set of trace files produced by a set of applications which was running on the computer system when the incident on the computer system occurred, (ii) a set of process images defining memory contents of a set of processes running the set of applications on the computer system, and (iii) a set of state snapshots identifying a set of current operating states of the computer system when the incident on the computer system occurred, and

storing the set of trace files, the set of process images, and the set of state snapshots in separate locations within the memory for individual searching using the diagnostics analyzers.

19. A computer program product as in claim 18 wherein the computer system includes multiple processing nodes connected together on a computer network;

and

wherein obtaining the electronic transmission from the computer system includes acquiring, from each processing node, (i) at least one trace file from that processing node, (ii) at least one process image from that processing node, and (iii) at least one state snapshot from that processing node.

20. A computer program product as in claim 19 wherein the multiple processing nodes of the computer system are constructed and arranged to operate as a data storage cluster;

and

wherein the method further comprises;

implementing a remedy on the data storage cluster based on the reason identified on the electronically generated report to cure operation of the data storage cluster.

21. A computer program product as in claim 20 wherein the data storage cluster resides at a field location, and the processing circuitry and the diagnostics analyzer database reside at an analyzer location which is separate from the field location;

and

wherein the method further comprises;

updating the diagnostics analyzer database residing at the analyzer location with new fault signature detector entries as new faults are discovered and new remedies are developed over time.

22. A computer program product as in claim 21, further comprising:

deleting outdated fault signature detector entries from the diagnostics analyzer database residing at the analyzer location as old faults and old remedies of the outdated fault signature entries become obsolete over time.