Rule based engines for diagnosing grid-based computing systems

• US 20060112061A1
• Filed: 11/22/2005
• Published: 05/25/2006
• Est. Priority Date: 06/24/2004
• Status: Abandoned Application

• Current Assignee: Sun Microsystems Incorporated (Oracle Corporation)
• Original Assignee: Sun Microsystems Incorporated (Oracle Corporation)
• Inventors: Masurkar, Vijay B.
• Application Number:
  US11/284,672
• Publication Number:
  US 20060112061A1
• Time in Patent Office: Days
• US Class Current: 706/47
• CPC Class Codes: G06F 11/0709 in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
• CPC Class Codes: G06F 11/0748 in a remote unit communicating with a single-box computer node experiencing an error/fault remote testing G06F11/2294
• CPC Class Codes: G06F 11/079 Root cause analysis, i.e. error or fault diagnosis in a hardware test environment G06F11/22; in a software test environment G06F11/36
• CPC Class Codes: G06N 5/025 Extracting rules from data
• CPC Class Codes: G06N 5/04 Inference or reasoning models

Abstract

Disclosed herein is the creation and utilization of autonomic agents that may be utilized on demand by service engineers to remotely diagnose and address faults, errors and other conditions within a grid-based computing system, and related computerized processes and network architectures and systems supporting such agents. The autonomic diagnostic agents can comprise software driven rules engines that operate on facts or data, such as telemetry and event information and data in particular, according to a set of rules. The autonomic diagnostic agents execute in accordance with the rules based on the facts and data found in the grid-based system, and then make a determination about the grid. The operations of a particular agent varies depending upon the status and configuration of the particular grid-based system being diagnosed as dictated by the database of rules. Particular memory allocations, diagnostic process and subprocess interactions, and rule constructs are disclosed.

Claims

1. A method for remotely diagnosing fault events in a grid-based computing system, the method comprising:

establishing a diagnostic metadata and rules database containing rules describing elements of and configuration aspects of said grid-based computing system, said rules comprising software objects;

establishing one or more diagnostic scripts each adapted to identify potential causes for particular fault events that may occur in said computing system, each said diagnostic script referencing said rules in said database to analyze metadata produced by said computing system;

receiving an indication of a fault event after it occurs in said computing system; and

initiating an autonomic diagnostic agent process in said computing system according to a diagnostic script associated with said occurred event, said autonomic diagnostic agent process comprising a rules-based engine that includes a diagnostic rules state machine subprocess and a diagnostic rules process controller subprocess, wherein said diagnostic rules state machine subprocess is adapted to load from the database into a diagnostic execution workspace the state-transition rules that establish how the engine moves between various diagnostic steps of the diagnostic script, and said diagnostic rules state machine subprocess is adapted to consider said loaded rules to perform appropriate diagnostic tasks as defined by said diagnostic steps of said associated diagnostic script, wherein said autonomic diagnostic agent process thereby provides an indication of a possible root cause for said occurred event in light of metadata obtained from said computing system.

2. The method of claim 1, wherein said indication of said occurred fault event is characterized by a fault monitoring subsystem of said computing system according to an established event framework, said framework defining events according to a plurality of event types and corresponding event sub-types for each event type and a corresponding severity level for said sub-types.

3. The method of claim 2, wherein said diagnostic scripts further describe event processors, said event processors being functions that are adapted to manage units of execution work upon the occurrence of an event as defined by said event framework, wherein said event processors include exception processors and synchronization processors.

4. The method of claim 1, wherein said rules being of types including:

i) diagnostic process rules defining procedures for diagnosing resources in said computing system;

ii) agent action rules relating to transitioning of steps for diagnosing said computing system, said agent action rules being used by said diagnostic rules process controller to define truth functions;

iii) granular diagnostic rules defining procedures for diagnosing finer components of said resources, wherein said diagnostic rules process controller subprocess considers one or more agent action rules for each diagnostic step; and

iv) foundation rules defining characteristics that apply to a particular family of resources;

wherein said resources include at least one of devices, subsystems, software, hardware and data structures of said computing system

5. The method of claim 4, wherein said truth functions are used by a diagnostic execution subprocess initiated by said diagnostic rules process controller subprocess to analyze one or more antecedents to determine an appropriate consequent, and said determined consequent affecting subsequent diagnostic tasks of said autonomic diagnostic agent.

6. The method of claim 4, wherein a diagnostic rules controller subprocess is invoked for said consideration of said agent action rules, said diagnostic rules controller subprocess consulting at least one agent action table corresponding to a particular relevant agent action rule.

7. The method of claim 1, wherein said indication of said occurred fault event is communicated by a fault monitoring subsystem of said computing system in a diagnostic event record, said diagnostic event record containing:

i) event data concerning said occurred fault and an associated resource in the computing system; and

ii) diagnostic telemetry information comprising data about the resource that experienced the event and concerning operation of the resource up to the occurrence of the fault event.

8. The method of claim 1, wherein said indication of a possible root cause for said occurred event comprises a derived list of suspected failed resources.

9. The method of claim 1, wherein said diagnostic rules state machine subprocess obtains necessary rules from said database after initialization of said autonomic diagnostic agent instance, said obtained rules being used by said diagnostic rules process controller to compile a web of states within diagnostic execution workspace, wherein said web of states comprises one or more state tables with each diagnostic step of said diagnostic script for said initiated autonomic diagnostic agent corresponding to a state table, said one or more state tables being maintained in cache memory so as to provide said diagnostic rules process controller subprocess with immediate access to rules relating to a current configuration of said computing system and said fault event, said web of states being customized to said initiated autonomic diagnostic agent.

10. The method of claim 9, wherein said diagnostic steps are invoked sequentially by said diagnostic rules process controller subprocess in accord with said web of states.

11. The method of claim 1, further comprising establishing a data center virtualization architecture in said grid-based computing system, said virtualization architecture including a grid diagnostic core in communication with a diagnostic management application, said application adapted to be accessible by a remote service engineer user to received fault event indications and initiate an autonomic diagnostic agent process, wherein said grid diagnostic core includes said database, a grid diagnostic telemetry interface, a telemetry configurator in communication with said telemetry interface and said database, a diagnostic kernel in communication with said telemetry configurator and said database, and a diagnostic service objects and methods library in communication with said diagnostic kernel, said diagnostic kernel adapted to spawn instances of autonomic diagnostic agent processes within said architecture.

12. A computer readable medium having computer readable code thereon for remotely diagnosing grid-based computing systems, the medium comprising:

instructions for establishing an electronically accessible diagnostic metadata and rules database containing rules describing elements of and configuration aspects of said grid-based computing system, said rules comprising software objects;

one or more diagnostic scripts each adapted to identify potential causes for particular fault events that may occur in said computing system, each said diagnostic script referencing said rules in said database to analyze metadata produced by said computing system;

instructions for receiving an indication of a fault event after it occurs in said computing system and displaying said fault to a user; and

instructions enabling said user to initiate an autonomic diagnostic agent process in said computing system according to a diagnostic script associated with said occurred event, said autonomic diagnostic agent process comprising a rules-based engine that includes a diagnostic rules state machine subprocess and a diagnostic rules process controller subprocess, wherein said diagnostic rules state machine subprocess is adapted to load from the database into a diagnostic execution workspace the state-transition rules that establish how the engine moves between various diagnostic steps of said associated diagnostic script, and said diagnostic rules state machine subprocess is adapted to consider said loaded rules to perform appropriate diagnostic tasks as defined by said diagnostic steps of said associated diagnostic script; and

wherein said autonomic diagnostic agent process thereby provides an indication of a possible root cause for said occurred event in light of metadata obtained from said computing system.

13. A grid-based computing system adapted to provide partially automated diagnosis of fault events, the computing system comprising:

a memory;

a processor;

a persistent data store;

a fault monitoring subsystem;

a communications interface; and

an electronic interconnection mechanism coupling the memory, the processor, the persistent data store, and the communications interface;

wherein said persistent data store contains a diagnostic metadata and rules database storing rules describing elements of and configuration aspects of said grid-based computing system, said rules comprising software objects, and said persistent data store further contains one or more diagnostic scripts each adapted to identify potential causes for particular fault events that may occur in said computing system, each said diagnostic script referencing said rules in said database to analyze metadata from said computing system;

wherein said fault monitoring subsystem is adapted to characterize an occurred fault event according to an established event framework, said framework defining events according to a plurality of event types and corresponding event sub-types for each event type and a corresponding severity level for said sub-types;

and wherein the memory is encoded with an application that when performed on the processor, provides a diagnostic process for processing information, the diagnostic process operating according to one of said diagnostic scripts and causing the computer system to perform the operations of;

receiving an indication of a fault event after it occurs in said computing system; and

initiating an autonomic diagnostic agent process in said computing system according to a diagnostic script associated with said occurred event, said autonomic diagnostic agent process comprising a rules-based engine that includes a diagnostic rules state machine subprocess and a diagnostic rules process controller subprocess, wherein said diagnostic rules state machine subprocess is adapted to load from the database into a diagnostic execution workspace the state-transition rules that establish how the engine moves between various diagnostic steps of the associated diagnostic script, and said diagnostic rules state machine subprocess is adapted to consider said loaded rules to perform appropriate diagnostic tasks as defined by said diagnostic steps of said associated diagnostic script, wherein said diagnostic scripts further describe event processors, said event processors being functions that are adapted to manage units of execution work upon the occurrence of an event as defined by said event framework; and

wherein said autonomic diagnostic agent process thereby provides an indication of a possible root cause for said occurred event in light of metadata obtained from said computing system.

14. The grid-based computing system of claim 13, wherein said and rules are of types including:

i) diagnostic process rules defining procedures for diagnosing resources in said computing system;

ii) agent action rules relating to transitioning of steps for diagnosing said computing system, said agent action rules being used by said diagnostic rules process controller to define truth functions;

iii) granular diagnostic rules defining procedures for diagnosing finer components of said resources, wherein said diagnostic rules process controller subprocess considers one or more agent action rules for each diagnostic step; and

iv) foundation rules defining characteristics that apply to a particular family of resources;

wherein said resources include at least one of devices, subsystems, software, hardware and data structures of said computing system.

15. The grid-based computing system of claim 14, wherein said truth functions are used by a diagnostic execution subprocess initiated by said diagnostic rules process controller subprocess to analyze one or more antecedents to determine an appropriate consequent, and said determined consequent affecting subsequent diagnostic tasks of said autonomic diagnostic agent.

16. The grid-based computing system of claim 14, wherein a diagnostic rules controller subprocess is invoked for said consideration of said agent action rules, said diagnostic rules controller subprocess consulting at least one agent action table corresponding to a particular relevant agent action rule.

17. The grid-based computing system of claim 13, wherein said diagnostic rules state machine subprocess obtains necessary rules from said database after initialization of said autonomic diagnostic agent instance, said obtained rules being used by said diagnostic rules process controller to compile a web of states within diagnostic execution workspace, wherein said web of states comprises one or more state tables with each diagnostic step of said diagnostic script for said initiated autonomic diagnostic agent corresponding to a state table, said one or more tables being maintained in cache memory so as to provide said diagnostic rules process controller subprocess with immediate access to rules relating to a current configuration of said computing system and said fault event, said web of states being customized to said initiated autonomic diagnostic agent.

18. The grid-based computing system of claim 17, wherein said diagnostic steps are invoked sequentially by said diagnostic rules process controller subprocess in accord with said web of states.

19. The grid-based computing system of claim 13, further comprising a data center virtualization architecture established in said grid-based computing system, said virtualization architecture including a grid diagnostic core in communication with a diagnostic management application, said application adapted to be accessible by a remote service engineer user to received fault event indications and initiate an autonomic diagnostic agent process.

20. The grid-based computing system of claim 19, wherein said grid diagnostic core includes said database, a grid diagnostic telemetry interface, a telemetry configurator in communication with said telemetry interface and said database, a diagnostic kernel in communication with said telemetry configurator and said database, and a diagnostic service objects and methods library in communication with said diagnostic kernel.