Multiple node/virtual input/output (I/O) server (VIOS) failure recovery in clustered partition mobility

• US 8,996,477 B2
• Filed: 09/15/2012
• Issued: 03/31/2015
• Est. Priority Date: 11/05/2010
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Mewhinney, Greg R., Nevarez, David, Pafumi, James A., Rosales, Jacob J.
• Primary Examiner(s): Reyes, Mariela
• Assistant Examiner(s): VU, THONG H
• Application Number:
  US13/621,207
• Publication Number:
  US 20130024718A1
• Time in Patent Office: 927 Days
• Field of Search: 707/692, 707/649, 707/2, 718/108, 718/1, 709/222
• US Class Current: 707/692
• CPC Class Codes: G06F 11/1425 by reconfiguration of node membership
• CPC Class Codes: G06F 11/1482 by means of middleware or OS functionality
• CPC Class Codes: G06F 11/203 using migration
• CPC Class Codes: G06F 11/2046 where the redundant components share persistent storage G06F11/2043 takes precedence
• CPC Class Codes: G06F 11/3006 where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems multiprogramming arrangements G06F9/46; allocation of resources G06F9/50
• CPC Class Codes: G06F 11/3055 Monitoring arrangements for monitoring the status of the computing system or of the computing system component, e.g. monitoring if the computing system is on, off, available, not available error or fault processing without redundancy G06F11/0703; error detection or correction by redundancy in data representation G06F11/08; error detection or correction by redundancy in operation G06F11/14; error detection or correction by redundancy in hardware G06F11/16
• CPC Class Codes: G06F 2201/815 Virtual middleware or OS functionality using virtual machines to implement generic software techniques for error detection or fault masking G06F11/1484

Abstract

A method utilizes cluster-awareness to effectively support a live partition mobility (LPM) event and provide recovery from node failure within a Virtual Input/Output (I/O) Server (VIOS) cluster. An LPM utility creates a monitoring thread on a first VIOS on initiation of a corresponding LPM event. The monitoring thread tracks a status of an LPM and records status information in the mobility table of a database. The LPM utility creates other monitoring threads on other VIOSes running on the (same) source server. If the first VIOS VIOS sustains one of multiple failures, the LPM utility provides notification to other functioning nodes/VIOSes. The LPM utility enables a functioning monitoring thread to update the LPM status. In particular, a last monitoring thread may perform cleanup/update operations within the database based on an indication that there are nodes on the first server that are in failed state.

Claims

1. In a cluster-aware data processing system having a processor, a memory coupled to the processor, at least one input/output (I/O) adapter that enables connection to an external network with a shared storage repository, and a plurality of virtual I/O servers (VIOSes) that form a VIOS cluster with a shared database, where each VIOS is cluster aware, a method comprising:

activating a first monitoring thread on a first VIOS of one or more VIOSes of a first server to track a status of a live partition mobility (LPM) event;

recording information about the LPM event within the shared database by using the first monitoring thread;

in response to the first VIOS sustaining a failure condition, identifying one or more functioning monitoring threads that continue to function on the first server, wherein the failure condition results in a loss of LPM event monitoring by the first monitoring thread;

determining whether the one or more functioning monitoring threads is a single, last monitoring thread; and

in response to receiving an indication that identifies at least the first VIOS of the one or more VIOSes on the first server as being in failed state, performing, via the last monitoring thread;

a query of a mobility table within the shared database to determine one or more failed VIOSes, including the first VIOS, within the one or more VIOSes of the first server that are in the failed state; and

a removal of one or more corresponding rows/entries that are associated with the one or more failed VIOSes from the mobility table.

2. The method of claim 1, wherein the activating a first monitoring thread further comprises:

detecting an initiation of the LPM event; and

in response to detecting the initiation of the LPM event, creating one or more of;

the first monitoring thread on the first VIOS on the first server, a first set of monitoring threads on other VIOSes of the one or more VIOSes on the first server, and (a second set of monitoring threads on each of one or more VIOSes on a second, target server.

3. The method of claim 1, wherein the recording further comprises:

recording information about the LPM event into a mobility table using the monitoring thread, wherein the recorded information includes information about one or more of;

a client;

the server; and

the status of a corresponding VIOS participating in the LPM event;

determining when the LPM event is terminated; and

in response to the LPM event being terminated, providing via the first monitoring thread one or more of;

associated updates to storage resources;

removal of a corresponding entry in a mobility table of the database; and

other maintenance operations.

4. The method of claim 1, further comprising:

detecting when the first VIOS has sustained the failure condition, wherein the first VIOS is associated with a virtualized client partition corresponding to the LPM;

in response to detecting that the first VIOS has sustained the failure condition and the first monitoring thread ceasing to exist, providing notification of the failed state of the first VIOS to a plurality of other functioning VIOSes in the VIOS cluster; and

enabling one or more of the plurality of other functioning VIOSes to appropriately update LPM event status in the database.

5. The method of claim 1, further comprising:

initiating continual checks about the state of the LPM event by other monitoring threads via communication of the other monitoring threads with a hypervisor;

determining whether the LPM event is one of;

terminated;

completed; and

in a failed state;

in response to the LPM event ending, providing to the other monitoring threads;

information that indicates that the LPM event is ended; and

information about a count of the set of other monitoring threads that currently monitors the LPM event; and

in response to the set of other monitoring threads being a single, last monitoring thread, initiating, via the last monitoring thread, a query of the database to determine whether there exists at least one VIOS of the one or more VIOSes that is in the failed state.

6. The method of claim 5, further comprising:

in response to the LPM event having ended as a result of a failure condition, returning to a particular stage of a corresponding migration process in order to resume and complete the mobility operation.

7. The method of claim 1, further comprising, in response to receiving an indication that identifies at least the first VIOS of the one or more VIOSes on the first server as being in failed state, performing, via the last monitoring thread:

a cleanup of one or more storage resources of the one or more failed VIOSes; and

one or more update operations within the shared database, wherein the one or more update operations are performed on a corresponding mobility table within the shared database and include one or more of;

operations needed to keep the shared database consistent and current and one or more maintenance operations.

8. The method of claim 1, wherein the failed state of the first VIOS is at least one condition from among:

the first VIOS no longer being capable of performing input/output (I/O) operations, the first VIOS going offline, and removal of the first VIOS.

9. The method of claim 1, further comprising:

in response to the first VIOS sustaining the failure condition, supplying the one or more functioning monitoring threads with a count of how many threads are currently monitoring the LPM event.