Knowledge based resource management

• US 5,675,724 A
• Filed: 03/25/1996
• Issued: 10/07/1997
• Est. Priority Date: 05/03/1991
• Status: Expired due to Term

• Current Assignee: Storage Technologies Corporation (Oracle Corporation)
• Original Assignee: Storage Technologies Corporation (Oracle Corporation)
• Inventors: Stephens, Michael Robert, Hill, Jo Etta, Jurestovsky, Nancy Rosann, Hill, Fletcher Lawrence, Beal, David Grant
• Primary Examiner(s): Baker, Stephen M.
• Application Number:
  US08/622,552
• Time in Patent Office: 561 Days
• Field of Search: 395/182.02
• US Class Current: 714/4.3
• CPC Class Codes: G05B 13/028 using expert systems only
• CPC Class Codes: G05B 23/0229 knowledge based, e.g. expert systems; genetic algorithms
• CPC Class Codes: G06F 11/0727 in a storage system, e.g. in a DASD or network based storage system drivers for digital recording or reproducing units G06F3/06; circuits for error detection or correction within digital recording or reproducing units G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097
• CPC Class Codes: G06F 11/079 Root cause analysis, i.e. error or fault diagnosis in a hardware test environment G06F11/22; in a software test environment G06F11/36
• CPC Class Codes: G06F 11/0793 Remedial or corrective actions recovery from an exception in an instruction pipeline G06F9/3861; by retry G06F11/1402; for recovering from a failure of a protocol instance or entity H04L69/40
• CPC Class Codes: G06F 11/2035 without idle spare hardware
• CPC Class Codes: G06F 11/2038 with a single idle spare processing component
• CPC Class Codes: G06F 11/2205 using arrangements specific to the hardware being tested
• CPC Class Codes: G06F 11/2257 using expert systems

Abstract

The knowledge based system identifies the occurrence of a failure within the customer equipment and functions using its rules, hypotheses and collected data to isolate the functional resource which is the source of the error in the customer equipment and, whenever possible, "fence" or isolate the failed functional resource that has caused the error. In addition, the knowledge based system makes use of resource related rules to remove a resource or set of resources available for concurrent other failure management operations. The resource related rules allow resources to function in any one non-operational or operational mode or to functional simultaneously in multiple non-operational modes.

Claims

1. Apparatus for controllably isolating operational elements in a customer system, which customer system includes a plurality of operational elements, each interconnected to at least one other operational element via an interconnection path, comprising:

means for storing data identifying all said operational elements and said interconnection paths;

means for detecting the presence of a failure in said associated customer system;

means for generating a failure report which contains data relevant to said detected failure, including an identification of the failure mode and, exclusively, all said operational elements that were cooperatively operative and active during said detected failure;

means for retrieving a portion of said data from said storing means to identify a subset of said operational elements and all said interconnection paths connected to said subset of operational elements; and

means for selectively disabling the use of said subset of operational elements, as identified by said retrieving means, by said customer system, comprising;

means for determining whether disabling said subset of operational elements reduces a number of available operational elements below a predetermined threshold; and

means, responsive to said number being above said threshold, for fencing said subset of operational elements by disabling any said interconnection paths connected to said subset of operational elements.

2. The apparatus of claim 1 further comprising:

means for storing said failure report;

means for maintaining a failure analysis history containing data identifying each of said operational elements and associated composite failure data representative of known operational element failures;

means for comparing said stored failure report with said failure analysis history;

means for identifying at least one failed operational element based on said comparison;

means, responsive to said identifying means, for writing data into said retrieving means to identify said at least one failed operational element and said interconnection paths connected to said identified failed operational element.

3. The apparatus of claim 1 further comprising:

means for identifying at least one of said identified active operational elements for testing;

means for verifying that said at least one operational element is not in use;

means, responsive to said at least one operational element not being in use, for identifying all said interconnection paths connected to said at least one operational element; and

means for writing data identifying said at least one operational element and said identified interconnection paths connected to said at least one operational element into said retrieving means to disable said at least one operational element.

4. The apparatus of claim 1, wherein said operational elements are capable of being activated in an online mode and a plurality of concurrently active offline fence modes, further comprising:

a plurality of means for activating ones of said fence modes on said operational elements contained in said subset of operational elements; and

wherein said selectively disabling means further comprises;

means for writing in said portion of data, indicia that identifies at least one fence mode in which each operational element in said subset of operational elements is disabled.

5. The apparatus of claim 4 wherein said selectively disabling means further comprises:

means, responsive to said apparatus requesting application of a fence mode to one of said subset of operational elements, for comparing said requested fence mode with said modes listed with said portion of said data;

means, responsive to said comparing means matching said requested fence mode with one of said listed fence modes, for accepting application of said requested fence mode to said requested operational element.

6. The apparatus of claim 1 further comprising:

means for storing said failure report;

means for maintaining a failure analysis history containing data identifying each of said operational elements and associated composite failure data representative of known operational element failures;

means for comparing said stored failure report with said failure analysis history;

means for generating a list of said operational elements which are common to a plurality of said failure reports;

means for writing data into said retrieving means to identify said common operational elements and said interconnection paths connected to said identified common operational elements; and

means for mapping said identified common operational elements to at least one failed field replaceable unit containing said common operational elements.

7. The apparatus of claim 6 wherein said mapping means comprises:

means for generating an alarm indication identifying said failed field replaceable unit.

8. A method for controllably isolating operational elements in a customer system, which customer system includes a plurality of operational elements, each interconnected to at least one other operational element via an interconnection path, comprising the steps of:

storing data in a memory to identify all said operational elements and said interconnection paths;

detecting the presence of a failure in said associated customer system;

generating a failure report which contains data relevant to said detected failure, including an identification of the failure mode and, exclusively, all operational elements that were cooperatively operative during said detected failure;

retrieving a portion of said data from said memory to identify a subset of said operational elements and all said interconnection paths connected to said subset of operational elements; and

selectively disabling the use of said subset of operational elements, as identified by said step of retrieving, by said customer system, comprising;

determining whether disabling said subset of operational elements reduces a number of available operational elements below a predetermined threshold; and

fencing, in response to said number being above said threshold, said subset of operational elements by disabling any said interconnection paths connected to said subset of operational elements.

9. The method of claim 8 further comprising the steps of:

storing said failure report;

maintaining a failure analysis history containing data identifying each of said operational elements and associated composite failure data representative of known operational element failures;

comparing said stored failure report with said failure analysis history;

identifying at least one failed operational element based on said comparison; and

writing in response to said identifying means, data into said memory to identify said at least one failed operational element and said interconnection paths connected to said identified failed operational element.

10. The method of claim 8 further comprising:

identifying at least one of said identified active operational elements for testing;

for verifying that said at least one operational element is not in use;

identifying in response to said at least one operational element not being in use, all said interconnection paths connected to said at least one operational element;

writing data identifying said at least one operational element and said identified interconnection paths connected to said at least one operational element into said memory to disable said at least one operational element.

11. The method of claim 8, wherein said operational elements are capable of being activated in an online mode and a plurality of concurrently active offline fence modes, said system including a plurality of processes for activating ones of said fence modes on said operational elements contained in said subset of operational elements and wherein said step of selectively disabling further comprises:

writing in said portion of data, indicia that identifies at least one fence mode in which each operational element in said subset of operational elements is disabled.

12. The method of claim 11 wherein said step of selectively disabling further comprises:

comparing in response to said apparatus requesting application of a fence mode to one of said subset of operational elements, said requested fence mode with said modes listed with said portion of said data;

accepting, in response to said step of comparing matching said requested fence mode with one of said listed modes, application of said requested fence mode to said requested operational element.

13. The method of claim 8 further comprising the steps of:

storing said failure report;

maintaining a failure analysis history containing data identifying each of said operational elements and associated composite failure data representative of known operational element failures;

comparing said stored failure report with said failure analysis history;

generating a list of said operational elements which are common to a plurality of said failure reports;

writing data into said memory to identify said common operational elements and said interconnection paths connected to said identified common operational elements; and

mapping said identified common operational elements to at least one failed field replaceable unit containing said common operational elements.

14. The method of claim 13 wherein said step of mapping comprises:

generating an alarm indication identifying said failed field replacement unit.