Apparatus and accompanying method for use in a sysplex environment for performing escalated isolation of a sysplex component in the event of a failure

• US 5,416,921 A
• Filed: 11/03/1993
• Issued: 05/16/1995
• Est. Priority Date: 11/03/1993
• Status: Expired due to Term

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Surman, David H., Nick, Jeffrey M., Swanson, Michael D., Shepard, William C., Frey, Jeffrey A., Meritt, Allan S., Goetze, Lisa M.
• Primary Examiner(s): ATKINSON, CHARLES
• Application Number:
  US08/147,351
• Time in Patent Office: 559 Days
• Field of Search: 395/575, 371/11.3, 364/269.2, 364/281.9, 364/285.3
• US Class Current: 714/11
• CPC Class Codes: G06F 11/2035 without idle spare hardware
• CPC Class Codes: G06F 11/2046 where the redundant components share persistent storage G06F11/2043 takes precedence

Abstract

Apparatus and accompanying methods for use in preferably a multi-system shared data (sysplex (5)) environment which quickly and efficiently isolates (fences), through a pre-defined hierarchical order, failed sysplex components from accessing shared data in order to protect data integrity. Specifically, by dividing a sysplex workload into specified fence groups (FG A, FG B) and providing appropriate software and hardware fence support, fencing can occur at various distinct levels: a member-to-member level, i.e. to allow any member (220, 225, 230, 233,237) of a fence group to fully isolate any other ("target") member of that same group; a fence group level, i.e. to isolate all members of a fence group that execute on a "target" system (200₁, 200₂, 200₃); and a system level, i.e. to fully isolate an entire "target" system. Through pre-defined escalation rules (630), fencing can be escalated from a lower member level to a higher, group or system, level in the event a lower level fence can not be successfully imposed. Member level fencing is accomplished in software (1300, 1500); group and system level fencing is accomplished through dedicated hardware fencing facilities (44). An identifier (444, 464) uniquely designates each different fence group existing on a computer processing complex (CPC) (40₁, 40₂, 40₃, 40_M) in the sysplex over the life of that CPC. Advantageously, this technique eliminates erroneous back level fencing, significantly expedites fence processing and also greatly reduces a need for human intervention.

Claims

1. In a computer having a resource shared by a plurality of processes, said processes executing on a plurality of different systems, a method for-isolating a failed component in said computer from said resource comprising the steps of:

in a first one of said plurality of different systems that executes a process in a fence group, wherein the fence group is defined as all processes, in said plurality of processes, that collectively and exclusively utilize said resource wherein each of said processes in the fence group executes on a corresponding one of said systems and is a different member of the group, the steps of;

issuing, whenever any one member in the group fails to properly execute on a second one of said systems, a fence request against the failed one member;

in said second one of the systems;

attempting, in response to said fence request, to impose a fence around said failed one member to completely isolate, through software, said member from utilizing the resource; and

if, as a result of said attempting step, the fence could not be imposed against the failed one member, escalating said fence, as specified in accordance with pre-defined escalation rules, to completely isolate either all members of said fence group which execute on said second system or said second system in its entirety from utilizing said resource.

2. The method in claim 1 further comprising the step, in collectively all of said systems that execute members of said fence group, of:

organizing all of said processes that collectively and exclusively utilize said resource into the fence group.

3. The method in claim 2 wherein said resource is pre-defined data.

4. The method in claim 3 wherein the first and second ones of the system are the same or different ones of said systems.

5. The method in claim 4 wherein said computer further comprises an input/output (I/O) system for storing information and a hardware fencing facility, said I/O system being responsive, through said fencing facility, to commands issued by each of said systems to access said information, said escalation step comprising the steps of:

consulting pre-defined escalation rules to determine whether said fence can be imposed at a fence group level for said failed one member, and

generating, if said consulting step reveals for the failed one member that a fence can be imposed at the fence group level, a single granular hardware fence request to said fencing facility to prevent any of said commands issued by the members of the fence group situated on the second system from reaching said I/O system.

6. The method in claim 5 wherein said I/O system has a function dataset commonly accessible to all of said systems and wherein said function dataset stores said escalation rules, said consulting step comprising the step of:

obtaining the escalation rules for said failed one member from the function dataset.

7. The method in claim 6 wherein said organizing step, for a given process executing on a third one of said systems, comprises the steps of:

if a desired fence group does not exist to encompass the given process, creating said desired fence group, through the third one of said systems, to encompass said given process as a member thereof, wherein said creating step comprises the steps of;

establishing a parameter value for said desired fence group, wherein the parameter value is unique over a lifetime of said computer; and

forming, using said parameter value, a unique association, within the third one of said systems, said function dataset and said fencing facility, that identifies all the members of said desired fence group to each of said systems;

or

if said desired fence group does exist, adding said given process as a member to said desired fence group, said adding step comprising the step of;

extending the association for the desired fence group, through the parameter value therefor, to encompass the given process as a member of the desired fence group.

8. The method in claim 7 wherein the fence group comprises a resource manager, as a member thereof, wherein said resource manager has its own fence member token and wherein the method comprises the step, in said resource manager, of performing I/O accesses or accesses through an external structured external storage device using said fence member token.

9. The method in claim 8 further comprising the step of in the resource manager, in the event said fence is imposed, preventing or quiescing all I/O and structured external storage device accesses on behalf of the fence group member being fenced.

10. The method in claim 7 wherein said third system further includes a software fence table and a hardware fence table, and said hardware fencing facility has an authorization vector, said software and hardware fence tables having corresponding pluralities of software and hardware fence table entries and said vector having a plurality of elements, wherein said organizing step further comprises the steps of:

assigning an available one of said software fence table entries to said given process;

said software fence table storing a different corresponding entry for each different member of every fence group executing on the third system;

if said given process member is a first member executing on said third system in the desired fence group;

assigning an available one of the hardware fence table entries to the desired fence group;

said hardware fence table storing a different corresponding entry for each different fence group that has a member which executes on the third system;

assigning an available one of the authorization vector elements to the desired fence group;

said authorization vector having a different corresponding element for each different fence group that has members which access the data through the fencing facility;

setting the available one software fence table entry to point to the available one hardware fence table entry; and

storing, into corresponding fields within the available one hardware fence table entry, both the parameter value for the desired fence group and an index value, said index value pointing to the available one authorization vector element; and

storing the parameter value for the desired fence group within the available one authorization vector element;

or

if said given process is not the first member executing on said third system in the desired fence group;

setting the available one software fence table entry to point to a corresponding hardware fence table entry previously assigned to the desired fence group.

11. The method in claim 10 wherein the function dataset also has a plurality of entries for fence group members, the association forming step further comprising the step of:

assigning an available one of the entries in the function dataset to the given process; and

setting said available one entry in the function dataset to specify said given process as a member of the fence group and provide the corresponding parameter value therefor; and

said organizing step further comprises the step of;

storing, if desired, escalation rules for said given process within the function dataset.

12. The method in claim 11 wherein said escalation rules for said given process specify whether escalation is permitted from a member level fence to a fence group level fence and from a fence group level fence to a system level fence, and whether the fence group level fence is to be skipped in favor of the system level fence.

13. The method in claim 10 wherein said attempting step comprises the steps of:

determining, in response to the fence request, whether a corresponding entry in the software fence table in said second one system is assigned to the failed one member and does not indicate that a fence has been imposed against said failed one member; and

if said corresponding entry is so assigned and does not indicate that the fence has been so imposed;

notifying an operating system component within said second one system to prevent any subsequent I/O requests from being communicated external to said operating system on behalf of said failed one member and to purge or complete all I/O requests on behalf of the failed one member that are then in progress or pending;

changing said corresponding entry in the software fence table for said second one system and a corresponding entry in the function dataset for the failed one member to both indicate that a fence has been imposed against said failed one member; and

notifying all remaining members of the fence group containing said failed one member that the fence has been imposed against said failed one member.

14. The method in claim 10 wherein said issuing step comprises the step of incorporating first and second fence member tokens into said fence request, said first fence member token identifying a member of the fence group, executing on said first system, which issues the fence request and said second fence member token identifying the failed one member of the fence group against which the fence is to be imposed.

15. The method in claim 10 further comprising, for each member of a fence group requesting access into said data, the steps of:

in a corresponding one of the systems within which said each member executes;

providing an I/O prevention identifier (IOPID) in conjunction with an I/O request in order to determine whether said access into the data can proceed;

said IOPID containing a pointer to a corresponding entry in a software fence table, situated in said corresponding one system, for said each member;

accessing an entry, in the software fence table situated within said corresponding one system, for the member identified by the IOPID; and

ascertaining from the accessed entry whether the member has been fenced or not and, in response thereto, further processing or failing the I/O request, respectively.

16. In a computer having a resource shared by a plurality of processes, said processes executing on a plurality of different systems, apparatus for isolating a failed component in said computer from said resource comprising:

in a first one of said plurality of different systems that executes a process in a fence group, wherein the fence group is defined as all processes, in said plurality of processes, that collectively and exclusively utilize said resource wherein each of said processes in the fence group executes on a corresponding one of said systems and is a different member of the group;

means for issuing, whenever any one member in the group fails to properly execute on a second one of said systems, a fence request against tho failed one member;

in said second one of the systems;

means for attempting, in response to said fence request to impose a fence around said failed one member to completely isolate, through software, said member from utilizing the resource; and

means for escalating the fence, as specified in accordance with pre-defined escalation rules, and if said attempting means could not impose the fence against the failed one member, to completely isolate either all members of said fence group which execute on said second system or said second system in its entirety from utilizing said resource.

17. The apparatus in claim 16 further comprising means for organizing all of said processes that collectively and exclusively utilize said resource into the fence group.

18. The apparatus in claim 17 wherein said resource is pre-defined data.

19. The apparatus in claim 18 wherein the first and second ones of the system are the same or different ones of said systems.

20. The apparatus in claim 19 wherein said computer further comprises:

a hardware fencing facility; and

an input/output (I/O) system for storing information, said I/O system being responsive through said fencing facility, to commands issued by each of said systems to access said information; and

wherein said escalation means comprises;

means for consulting pre-defined escalation rules to determine whether said fence can be imposed at a fence group level for said failed one member, and

means for generating, if said consulting means reveals for the failed one member that a fence can be imposed at the fence group level, a single granular hardware fence request to said fencing facility to prevent any of said commands issued by the members of the fence group situated on the second system from reaching said I/O system.

21. The apparatus in claim 20 wherein said I/O system comprises a function dataset commonly accessible to all of said systems and wherein said function dataset stores said escalation rules, and wherein said consulting means comprises means for obtaining the escalation rules for said failed one member from the function dataset.

22. The apparatus in claim 21 wherein said organizing means comprises:

means for creating a desired fence group to encompass a given process as a member thereof and executing on a third one of said systems if said desired fence group does not exist wherein said creating means comprises, on said third system;

means for establishing a parameter value for said desired fence group, wherein the parameter value is unique over a lifetime of said computer; and

means for forming, using said parameter value, a unique association, within the third one of said systems, said function dataset and said fencing facility, that identifies all the members of said desired fence group to each of said systems; and

means for adding said given process as a member to said desired fence group, if said desired fence group already exists, by extending the association for the desired fence group, through the parameter value therefor, to encompass the given process as a member of the desired fence group.

23. The apparatus in claim 22 wherein the fence group comprises a resource manager, as a member thereof, wherein said resource manager has its own fence member token and performs I/O accesses or accesses through an external structured external storage device using said fence member token.

24. The apparatus in claim 23 wherein the resource manager, in the event said fence is imposed, prevents or quiesces all I/O and structured external storage device accesses on behalf of the fence group member being fenced.

25. The apparatus in claim 22 wherein said third system further comprises:

a software fence table;

a hardware fence table; and

wherein said hardware fencing facility has an authorization vector, said software and hardware fence tables having corresponding pluralities of software and hardware fence table entries and said vector having a plurality of elements; and

wherein said organizing means further comprises;

means for assigning an available one of said software fence table entries to said given process;

said software fence table storing a different corresponding entry for each different member of every fence group executing on the third system;

means, if said given process member is a first member executing on said third system in the desired fence group, for;

assigning an available one of the hardware fence table entries to the desired fence group;

said hardware fence table storing a different corresponding entry for each different fence group that has a member which executes on the third system;

assigning an available one of the authorization vector elements to the desired fence group;

said authorization vector having a different corresponding element for each different fence group that has members which access the data through the fencing facility;

setting the available one software fence table entry to point to the available one hardware fence table entry;

storing, into corresponding fields within the available one hardware fence table entry, both the parameter value for the desired fence group and an index value, said index value pointing to the available one authorization vector element; and

storing the parameter value for the desired fence group within the available one authorization vector element; and

means, if said given process is not the first member executing on said third system in the desired fence group, for setting the available one software fence table entry to point to a corresponding hardware fence table entry previously assigned to the desired fence group.

26. The apparatus in claim 25 wherein the function dataset also has a plurality of entries for fence group members, the association forming means further comprising:

means for assigning an available one of the entries in the function dataset to the given process; and

means for setting said available one entry in the function dataset to specify said given process as a member of the fence group and provide the corresponding parameter value therefor; and

said organizing means further comprises;

means for storing escalation rules for said given process within the function dataset.

27. The apparatus in claim 26 wherein said escalation rules for said given process specify whether escalation is permitted from a member level fence to a fence group level fence and from a fence group level fence to a system level fence, and whether the fence group level fence is to be skipped in favor of the system level fence.

28. The apparatus in claim 25 wherein said attempting means comprises:

means for determining, in response to the fence request, whether a corresponding entry in the software fence table in said second one system is assigned to the failed one member and does not indicate that a fence has been imposed against said failed one member; and

means, if said corresponding entry is so assigned and does not indicate that the fence has been so imposed, for;

notifying an operating system component within said second one system to prevent any subsequent I/O requests from being communicated external to said operating system on behalf of said failed one member and to purge or complete all I/O requests on behalf of the failed one member that are then in progress or pending;

changing said corresponding entry in the software fence table for said second one system and a corresponding entry in the function dataset for the failed one member to both indicate that a fence has been imposed against said failed one member; and

notifying all remaining members of the fence group containing said failed one member that the fence has been imposed against said failed one member.