Computer system fault recovery based on historical analysis

• US 4,633,467 A
• Filed: 07/26/1984
• Issued: 12/30/1986
• Est. Priority Date: 07/26/1984
• Status: Expired due to Term

• Current Assignee: Bell Telephone Laboratories, Inc. (Nokia Corporation)
• Original Assignee: AT&T, Inc.
• Inventors: Abel, Mark J., Kwan, Chi L., Veach, Michael T., Polli, Philip V.
• Primary Examiner(s): Smith, Jerry
• Assistant Examiner(s): Beausoliel, Jr., Robert W.
• Application Number:
  US06/634,461
• Time in Patent Office: 887 Days
• Field of Search: 371/16, 371/11, 371/9, 364/200
• US Class Current: 714/45
• CPC Class Codes: G06F 11/0715 in a system implementing multitasking multitasking per se G06F9/46
• CPC Class Codes: G06F 11/079 Root cause analysis, i.e. error or fault diagnosis in a hardware test environment G06F11/22; in a software test environment G06F11/36
• CPC Class Codes: G06F 11/20 using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements

Abstract

A method of identifying faulty units in a computer-controlled system. The system units generate error reports in response to the detection of error conditions. When an error report is received, an initial list is generated containing probable fault weights for each of the system units based on the type of the error report. The probable fault weights are prespecified based on a logical analysis of the fault modes and error propagation paths in the system. A history list of fault weights is first aged and then combined with the initial list to generate a resultant list. The resultant list becomes the new history list. The resultant list is then masked by the initial list to form a selection list from which a most probable fault unit is selected.

Claims

1. In a computer system comprising a processing unit (CPU), a peripheral unit (SM), and at least one interface unit (MSCU, MMP, MICU, TMS, DLI) connecting the processing unit to the peripheral unit, and in which prescribed ones of the units include means for generating and returning error report messages to the processing unit in response to detections of prescribed error conditions, a software implemented method for controlling the processing unit to identify a faulty system unit, said method comprising the steps of

in response to receipt of an error report identifying a predefined error condition and one of the prescribed units generating the error report, generating an initial list of predetermined fault probability weights for the system units,

aging a history list containing fault weights generated by the next-mentioned step as a result of receipt of a last error report by reducing the fault weights for each unit in the history list according to a first prescribed algorithm based on elapsed time since receipt of the last error report,

generating a new history list by individually combining the fault weights in the initial list with the fault weights in the aged history list for each unit according to a second prescribed algorithm, and

selecting as faulty unit that unit having the largest fault weight in the new history list that also has a non-zero fault weight in the initial list.

2. The method of claim 1 wherein prescribed ones of the units are replicated and reconfigurable to form a plurality of alternative communication paths between the processing unit and the peripheral unit and the step of generating an initial probability weight list further comprises the steps of

identifying a suspect one of the communication paths containing the specific unit that generated the error report,

identifying all of the remaining individual units in the suspect path, and

assigning individual predetermined fault probability weights to each of the units in the suspect path based on the type of the error report received.

3. The method of claim 1 wherein the step of aging the history list further comprises the step of

reducing the fault probability weight for each unit in the history list by an amount equal to a prescribed constant multiplied by a representation of the time that has elapsed since receipt of the last error report.

4. The invention of claim 3 in which said representation of elapsed time is the number of seconds elapsed since receipt of the last error report.

5. The method of claim 1 wherein the step of generating a new history list further comprises the step of

adding the corresponding weights for each unit from the present history list and the initial list.

6. The invention of claim 1 wherein the step of selecting a fault unit further comprises the step of

generating a selection list including only units common to both the initial list and the new history list and containing fault weights for the units contained therein taken from the new history list.

7. The invention of claim 1 further comprising the steps of

in response to a first error report of a prescribed type generated by a predetermined one of the units,

activating a prescribed timing interval, and

in response to another error report of the prescribed type generated by the predetermined one of the units during the prescribed timing interval,

selecting a prespecified another one of the system units as faulty if a prescribed threshold is exceeded.

8. The invention of claim 7 wherein the predetermined one of the units is a member of a replicated set of the units and the prescribed threshold is established by the step of

determining the difference between the number of units in the replicated set of units that are available for service at the time of receipt of the first error report and the number of units in the replicated set of units that are available for service at the time of receipt of said another one of the error reports of the prescribed type.

9. The invention of claim 8 wherein the step of selecting a prespecified another one of the system units as faulty further comprises the step of selecting a unit that interfaces with the set of replicated units.

10. The invention of claim 9 wherein the prescribed threshold is further established by the step of

comparing said difference to a threshold number obtained from a prestored table, said threshold number being obtained by

searching the table for a first entry containing a number representing the number of units in the replicated set of units that are available for service at the time of receipt of the first error report of the prescribed type, and

comparing said difference to a number contained in a second entry of the prestored table associated with the first entry.

11. The invention of claim 1 further comprising the steps of

performing diagnostic tests on the unit selected as faulty to confirm the fault condition, and

in response to an all test pass condition of the diagnostic tests, reducing the fault weight for the selected unit in the history list by a prescribed amount, but to an amount not less than zero.

12. In a computer system comprising replicated processing units, a peripheral unit, and at least one set of replicated interface units connecting the processing units to the peripheral unit via alternative communication paths, and in which each unit includes means for detecting a plurality of different error conditions signifying fault conditions and for generating and returning unique error reports to the processing unit in response to the detection of the different error conditions, a software implemented method for identifying a faulty unit, said method comprising the steps of

in response to a receipt of an error report of a given type,

determining the specific unit generating the error report,

determining an active communication path containing the specific unit,

determining all other units contained in the active communication path,

obtaining predefined probable fault weights for each type of unit in the system based on the error report type,

generating an initial list of suspect units by assigning the predefined probable fault weights for each type of system unit to the corresponding units in the active communication path,

obtaining an existing history list containing present fault weights for each of the system units,

aging the history list by reducing the fault weights for each unit according to a second prescribed algorithm based on time elapsed since receipt of a last error report,

generating a new history list by logically combining the initial list and the present history list in a predetermined manner, and

selecting as a fault unit that unit having the largest fault weight in the new history list that also has a non-zero fault weight in the initial list.