System and method to detect errors and predict potential failures

• US 7,409,594 B2
• Filed: 07/06/2004
• Issued: 08/05/2008
• Est. Priority Date: 07/06/2004
• Status: Expired due to Fees

• Current Assignee: Intel Corporation
• Original Assignee: Intel Corporation
• Inventors: Mukherjee, Udayan, Kundu, Aniruddha
• Primary Examiner(s): DUNCAN, MARC M
• Application Number:
  US10/887,368
• Publication Number:
  US 20060010352A1
• Time in Patent Office: 1,491 Days
• Field of Search: 714/26
• US Class Current: 714/26
• CPC Class Codes: G06F 11/0724 in a multiprocessor or a multi-core unit multiprocessors per se G06F15/80
• CPC Class Codes: G06F 11/076 by exceeding a count or rate limit, e.g. word- or bit count limit
• CPC Class Codes: G06F 11/0775 Content or structure details of the error report, e.g. specific table structure, specific error fields
• CPC Class Codes: G06F 11/079 Root cause analysis, i.e. error or fault diagnosis in a hardware test environment G06F11/22; in a software test environment G06F11/36
• CPC Class Codes: G06F 11/0793 Remedial or corrective actions recovery from an exception in an instruction pipeline G06F9/3861; by retry G06F11/1402; for recovering from a failure of a protocol instance or entity H04L69/40

Abstract

A system is disclosed that includes a component, a fault table configured to receive fault information associated with the component, and a diagnosis processor configured to read the fault information from the fault table and initiate corrective action as a function of the fault information. A method for handling faults in the system is also disclosed.

Claims

1. A system, comprising:

a component;

a fault table configured to receive fault information associated with the component and a time value associated with the fault information; and

a diagnosis processor configured to read the fault information from the fault table and initiate corrective action as a function of the fault information and a time the fault information was received, wherein the fault information is removed from the fault table when the time value associated with the fault information indicates the fault information is older than a predetermined threshold.

2. The system of claim 1, wherein the fault table is contained in the diagnosis processor.

3. The system of claim 1, wherein the fault table is contained outside the diagnosis processor.

4. The system of claim 1, wherein the fault table includes an entry, said entry including a field indicative of the identity of the component.

5. The system of claim 1, wherein the fault table includes an entry, said entry including a time value field.

6. The system of claim 5, wherein the time value field is indicative of when a fault was detected in the component.

7. The system of claim 5, wherein the time value field is indicative of the time that has elapsed since a fault was detected in the component.

8. The system of claim 1, wherein the fault table includes an entry, said entry including information indicative of the nature of a fault that has been detected in the component.

9. The system of claim 1, wherein the fault table includes an entry, said entry including a leaky bucket fault counter.

10. The system of claim 1, wherein the diagnosis processor is to predict a failure of the component.

11. The system of claim 1, wherein the component includes a fault register. and the fault information includes a value read from the fault register.

12. The system of claim 1, further comprising a satellite diagnosis processor in communication with the diagnosis processor, the satellite diagnosis processor configured to collect fault data from the component.

13. The system of claim 12, wherein the satellite diagnosis processor is located on a separate circuit board than the diagnosis processor.

14. The system of claim 12, wherein the satellite diagnosis processor is configured to preprocess the fault data, and transmit the preprocessed fault data to the diagnosis processor.

15. The system of claim 14, wherein the fault information associated with the component is a function of the fault data collected from the component by the satellite diagnosis processor.

16. The system of claim 1, further comprising, a chassis management processor in communication with the diagnosis processor. the chassis management processor configured to collect chassis fault data.

17. The system of claim 16, wherein the chassis fault data includes temperature information.

18. The system of claim 16, wherein the chassis fault data includes cooling device operating state information.

19. The system of claim 16, wherein the chassis fault data includes power operating state information.

20. The system of claim 1, wherein the corrective action includes initiating a diagnostic procedure for the component.

21. The system of claim 20, wherein the component includes a built-in self test, and the diagnostic procedure includes executing the built-in self test.

22. The system of claim 1, wherein the corrective action includes disabling the component.

23. The system of claim 1, wherein the corrective action includes replacing the component by a spare component.

24. The system of claim 1, wherein the corrective action includes generating an alert message.

25. The system of claim 1, wherein the component includes a memory.

26. The system of claim 25, wherein the fault information includes an indication that a parity error occurred in the memory.

27. The system of claim 25, wherein the fault information includes a count of the number of parity errors in a predetermined time interval.

28. The system of claim 1, wherein the component includes a disk drive controller.

29. The system of claim 28, wherein the fault information includes an indication that a retry occurred during an I/O operation to the component.

30. The system of claim 28, wherein the fault information includes a count of the number of I/O errors in a predetermined time interval.

31. The system of claim 1, wherein the fault information includes a value of a watchdog timer associated with the component.

32. The system of claim 1, wherein the fault information includes a leaky bucket fault counter associated with the component.

33. The system of claim 1, wherein the diagnosis processor is to predict a failure of the component using a policy.

34. The system of claim 1, wherein the diagnosis processor is to predict a failure of the component based on whether more than a predetermined number of faults in a given time window has occurred.

35. The system of claim 34, wherein the time window is modified based on a frequency of faults in said component.

36. One of a diagnosis processor and a satellite diagnostic processor, comprising:

a fault information table configured to store fault information associated with components of a system and a time value associated with the fault information; and

a processor in communication with the fault information table, the processor configured to analyze the fault information stored in the fault information table, wherein the processor is configured to select corrective actions based on analysis of the fault information stored in the fault information table and a time the fault information was received, wherein the fault information is removed from the fault information table when the time value associated with the fault information indicates the fault information is older than a predetermined threshold.

37. A method for handling faults in a system, comprising:

receiving fault information associated with a component;

storing a time value with the fault information in a fault information table;

taking corrective action as a function of the fault information and a time the fault information was received; and

removing the fault information from the fault information table when the time value associated with the fault information indicates the fault information is older than a predetermined threshold.

38. The method of claim 37, further comprising:

predicting the likelihood of failure of the component based on the fault information in the fault information table.

39. The method of claim 38 wherein predicting the likelihood of failure of the component is also based on a policy.

40. The method of claim 38 wherein predicting the likelihood of failure of the component is based on whether more than a predetermined number of faults in a given time window has occurred.

41. The method of claim 40, further comprising modifying the time window based on a frequency of faults in said component.

42. An article of manufacture comprising a computer-readable medium having stored thereon instructions adapted to be executed by a processor the instructions which, when executed, define a series of steps to be used to control a method for handling faults in a system, said steps comprising:

receiving fault information associated with a component;

storing a time value with the fault information in a fault information table;

taking corrective action as a function of the fault information and a time the fault information was received; and

removing the fault information from the fault information table when the time value associated with the fault information indicates the fault information is older than a predetermined threshold.

43. The article of manufacture of claim 42, wherein said steps further comprise:

predicting the likelihood of failure of the component based on the fault information in the fault information table.

44. A method for handling faults in a system, comprising:

receiving fault information associated with a component;

storing the fault information in a fault information table;

predicting the likelihood of failure of the component based on the fault information in the fault information table and whether more than a predetermined number of faults in a given time window has occurred;

modifying the time window based on a frequency of faults in the component; and

taking corrective action as a function of the fault information and a time the fault information was received.

45. An article of manufacture comprising a computer-readable medium having stored thereon instructions adapted to be executed by a processor, the instructions which, when executed, define a series of steps to be used to control a method for handling faults in a system, said steps comprising:

receiving fault information associated with a component;

storing the fault information in a fault information table;

predicting the likelihood of failure of the component based on the fault information in the fault information table and whether more than a predetermined number of faults in a given time window has occurred;

modifying the time window based on a frequency of fault in the component; and

taking corrective action as a function of the fault information and a time the fault information was received.

46. A system, comprising:

a component;

a fault table configured to receive fault information associated with the component; and

a diagnosis processor configured to read the fault information from the fault table and initiate corrective action as a function of the fault information, configured to predict the likelihood of failure of the component based on the fault information in the fault table and whether more than a predetermined number of faults in a given time window has occurred, and configured to modify the time window based on a frequency of faults in the component.