Fault tolerant system shared system resource with state machine logging

• US 7,219,260 B1
• Filed: 05/26/2000
• Issued: 05/15/2007
• Est. Priority Date: 05/26/2000
• Status: Expired due to Term

• Current Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Original Assignee: EMC Corporation (Dell Technologies Inc.)
• Inventors: de Forest, Miles Aram, MacHardy, Earle Trounson Jr.
• Primary Examiner(s): Bonzo; Bryce P.
• Assistant Examiner(s): MASKULINSKI, MICHAEL C
• Application Number:
  US09/579,670
• Time in Patent Office: 2,545 Days
• Field of Search: 714/15, 714/17, 714/20, 714/23, 711/161, 711/162, 709/100, 709/101, 709/312, 707/10, 707/202
• US Class Current: 714/15
• CPC Class Codes: G06F 11/1471 involving logging of persistent data for recovery
• CPC Class Codes: G06F 11/1474 in transactions G06F16/20 takes precedence
• CPC Class Codes: G06F 11/2056 by mirroring

Abstract

A system and method for logging and restoring the state of execution of resource transactions in a shared system resource by logging and restoration of state machine information. Each state machine logging mechanism includes a state machine log generator for extracting state machine information and generating and storing a state machine log. The resource may include dual, concurrently operating control/processing subsystems and may include corresponding functionally integral but physically separate state machine log mirroring mechanisms for receiving and restoring mirror copies of the state machine information.

Claims

1. A file server performing file transaction operations in response to file transaction requests by the clients and including a state machine logging mechanism, comprising:

a storage sub-system, and

a control/processing sub-system including

a file system processor performing file transaction operations in response to client requests and controlling file storage operations of the storage sub-system, and

a state machine logging mechanism operating concurrently and cooperative with the file system processor, including

a state machine log generator for extracting sequential state machine information defining a corresponding sequence of state machines during an execution of an operation, the sequence of state machines representing sequential states of execution of a file transaction wherein a state machine is comprised of state information including control and data values representing a state of operation of the control/processing sub-system during a corresponding step in the execution of the file transaction, wherein

the control/processing sub-system is a state machine system defined during the execution of a step in the execution of an operation by a sequential state machine defined by a state of operation of the state machine system during the step in the execution of the operation, and wherein

a file transaction operation is represented by a sequence of state machines wherein each state machine is defined by data and control values residing in the state machine system during existence of a state machine of the sequence, and

a state machine log for storing the state machine information, wherein the state machine log generator is responsive to the restoration of operation of the file server after a failure of file server operations for reading the state machine information from the state machine log and restoring the state of execution of a file transaction.

2. The file server of claim 1, wherein the state machine logging mechanism further comprises:

a state machine log mirroring mechanism that is functionally integral with the state machine log and that operates concurrently and in parallel with the state machine log and that is located separately from the control/processing sub-system and communicating with the state machine log generator for receiving and storing mirror copies of the state machine information, wherein the state machine log mirroring mechanism is responsive to the restoration of operation of the file server after a failure of file server operations for reading the state machine information from the state machine log mirroring mechanism and restoring the state of execution of a file transaction.

3. A file server performing file transaction operations in response to file transaction requests by the clients and including a state machine logging mechanism, comprising:

a storage sub-system, and

first and second control/processing sub-systems operating concurrently and in parallel, each including a file system processor performing file transaction operations in response to client requests directed to the first and second control/processing sub-systems and controlling file storage operations of the storage sub-system, and

a state machine logging mechanism operating concurrently and cooperatively with the file system processor, including a state machine log generator for extracting state machine information defining a sequence of state machines during an execution of an operation, each state machine representing a current state of execution of a step in the execution of a file transaction by the corresponding control/processing sub-system wherein a state machine is comprised of state information including control and data values representing a state of operation of the control/processing sub-system during a corresponding step in the execution of a file transaction, wherein

each control/processing sub-system is a state machine system defined during the execution of an operation by a sequence of state machines defined by states of operation of the state machine system during corresponding steps in the execution of the operation, and wherein

a file transaction operation is represented by at least a sequence of state machines wherein each state machine is defined by data and control values residing in the state machine system during existence of state machine of the sequence, and

a state machine log for storing the state machine information of the corresponding control/processing sub-system, wherein the state machine log generator is responsive to the restoration of operation of the file server after a failure of the corresponding control/processing sub-system for reading the state machine information from the corresponding state machine log and restoring the state of execution of a file transaction of the corresponding control/processing sub-system.

4. The file server of claim 3, wherein each control/processing sub-system further comprises:

a state machine log mirroring mechanism that is functionally integral with the state machine log and that operates concurrently and in parallel with the state machine log and that is located separately from the state machine log mechanism and that is communicating with the state machine log generator of the other control/processing sub-system for receiving and storing copies of the state machine information of the other control/processing sub-system, wherein the state machine log mirroring mechanism is responsive to the restoration of operation of the other control/processing sub-system after a failure of the other control/processing sub-system for reading the state machine information from the state machine log mirroring mechanism to the other control/processing sub-system and restoring the state of execution of a file transaction of the other control/processing sub-system of execution of a file transaction of the other control/processing sub-system.

5. A system resource performing system resource operations in response to requests by the clients and including a state machine logging mechanism, comprising:

a system resource-sub-system, and

a control/processing sub-system including a resource control processor performing system resource operations in response to client requests and controlling operations of the system resource sub-system, and

a state machine logging mechanism operating concurrently and cooperatively with the control/processing sub-system, including a state machine log generator for extracting state machine information defining a sequence of state machines during an execution of an operation, each state machine representing a current state of execution of a corresponding step in the execution of a system resource operation wherein a state machine is comprised of state information including control and data values representing a state of operation of the control/processing sub-system during a corresponding step in the execution in the system resource operation, wherein

the control/processing sub-system is a state machine system defined during the execution of an operation by a sequence of state machines defined by corresponding states of operation of the state machine system during corresponding steps in the execution of the operation, and wherein

a resource operation is represented by a sequence of state machines wherein each state machine is defined by data and control values residing in the state machine system during existence of a state machine of the sequence, and

a state machine log for storing the state machine information, wherein the state machine log generator is responsive to the restoration of operation of the system resource after a failure of system resource operations for reading the state machine information from the state machine log and restoring the state of execution of a system resource operation.

6. The system resource of claim 5, wherein the state machine logging mechanism further comprises:

a state machine log mirroring mechanism that is functionally integral with the state machine log and that operates concurrently and in parallel with the state machine log and that is located separately from the control/processing sub-system and communicating with the state machine log generator for receiving and storing mirror copies of the state machine information, wherein the state machine log mirroring mechanism is responsive to the restoration of operation of the system resource after a failure of system resource operations for reading the mirror copies of the state machine information from the state machine log mirroring mechanism and restoring the state of execution of a system resource operation.

7. A system resource performing system resource operations in response to system resource requests by the clients and including a state machine logging mechanism, comprising:

a system resource sub-system, and

first and second control/processing sub-systems operating concurrently and in parallel, each including a system processor performing system resource operations in response to client requests directed to the first and second control/processing sub-systems and controlling operations of the system resource sub-system, and

a state machine logging mechanism operating concurrently and cooperatively with the control/processing sub-system, including a state machine log generator for extracting state machine information defining a sequence of state machines during an execution of an operation, each state machine representing a state of execution of a corresponding step in the execution of a system resource operation of the corresponding control/processing sub-system wherein a state machine is comprised of state information including control and data values representing a state of operation of the control/processing sub-system during the corresponding step in the execution of the system resource operation, wherein

each control/processing sub-system is a state machine system defined during the execution of an operation by a sequence of state machines defined by a state of operation of the state machine system during a step in the execution of the operation, and wherein

a system resource operation is represented by a sequence of state machines wherein each state machine is defined by data and control values residing in the state machine system during existence of a state machine of the sequence of state machines, and

a state machine log for storing the state machine information of the corresponding control/processing sub-system, wherein the state machine log generator is responsive to the restoration of operation of the system resource after a failure of the corresponding control/processing sub-system for reading the state machine information from the corresponding state machine log and restoring the state of execution of a system resource operation of the corresponding control/processing sub-system.

8. The system resource of claim 7, wherein each control/processing sub-system further comprises:

a state machine log mirroring mechanism that is functionally integral with the state machine log and that concurrently and in parallel with the state machine log and that is located separately from the state machine log mechanism and that is communicating with the state machine log generator of the other control/processing sub-system for receiving and storing mirror copies of the state machine information of the other control/processing sub-system, wherein the state machine log mirroring mechanism is responsive to the restoration of operation of the other control/processing sub-system after a failure of the other control/processing sub-system for reading the mirror copies of the state machine information from the state machine log mirroring mechanism to the other control/processing sub-system and restoring the state of execution of a system resource operation of the other control/processing sub-system.

9. A state machine logging mechanism for use in a system resource performing system resource operations in response to requests by the clients, the system resource including a system resource sub-system and a control/processing sub-system including a resource control processor performing system resource operations in response to client requests and controlling operations of the system resource sub-system, the state machine logging mechanism comprising:

a state machine log generator operating concurrently and cooperatively with the control/processing sub-system for extracting state machine information defining a sequence of state machines during an execution of an operation, state machine representing a current state of execution of a step in the execution of a system resource operation wherein a state machine is comprised of state information including control and data values representing a state of operation of the control/processing sub-system during execution of the corresponding step in the execution of the system resource operation, and

a state machine log for storing the state machine information, wherein the state machine log generator is responsive to the restoration of operation of the system resource after a failure of system resource operations for reading the state machine information from the state machine log and restoring the state of execution of a system resource operation wherein

the control/processing sub-system is a state machine system defined during the execution of an operation by a sequence of state machines defined by corresponding states of operation of the state machine system during a step in the execution of the operation, and wherein

a file transaction operation is represented by a sequence of state machines wherein each state machine is defined by data and control values residing in the state machine system during existence of the corresponding state machine of the sequence of state machines.

10. The state machine logging mechanism of claim 9, further comprising:

a state machine log mirroring mechanism that is functionally integral with the state machine log and operating concurrently and in parallel with the state machine log and that is located separately from the control/processing sub-system and communicating with the state machine log generator for receiving and storing mirror copies of the state machine information, wherein the state machine log mirroring mechanism is responsive to the restoration of operation of the system resource after a failure of system resource operations for reading the mirror copies of the state machine information from the state machine log mirroring mechanism and restoring the state of execution of a system resource operation.

11. A state machine logging mechanism for use in a system resource performing system resource operations in response to system resource requests by the clients, the system resource including a system resource sub-system and first and second control/processing sub-systems, each including a system processor performing system resource operations in response to client requests directed to the first and second control/processing sub-systems and controlling operations of the system resource sub-system, the state machine logging mechanism comprising:

in each control/processor sub-system,

a state machine log generator for extracting state machine information defining a sequence of states machine during an execution of an operation, each state machine representing a state of execution of a step in the execution of a system resource operation of the corresponding control/processing sub-system wherein a state machine is comprised of state information including control and data values representing a state of operation of the control/processing sub-system during execution of the corresponding step in the execution of an operation, and

a state machine log operating concurrently and cooperatively with the corresponding control/processing sub-system for storing the state machine information of the corresponding control/processing sub-system, wherein the state machine log generator is responsive to the restoration of operation of the system resource after a failure of the corresponding control/processing sub-system for reading the state machine information from the corresponding state machine log and restoring the state of execution of a system resource operation of the corresponding control/processing sub-system, wherein

each control/processing sub-system is a state machine system defined during the execution of an operation by a sequence of state machines defined by a state of operation of the state machine system during a step in the execution of the operation, and wherein

a file transaction operation is represented by a sequence of states machine wherein each state machine is defined by data and control values residing in the state machine system during existence of the state machine of the sequence of state machines.

12. The state machine logging mechanism of claim 11, further comprising:

in each control/processor sub-system,

a state machine log mirroring mechanism that is functionally integral with the state machine log and that operates concurrently and in parallel with the state machine log and that is located separately from the state machine log mechanism and that is communicating with the state machine log generator of the other control/processing sub-system for receiving and storing mirror copies of the state machine information of the other control/processing sub-system, wherein the state machine log mirroring mechanism is responsive to the restoration of operation of the other control/processing sub-system after a failure of the other control/processing sub-system for reading the mirror copies of the state machine information from the state machine log mirroring mechanism to the other control/processing sub-system and restoring the state of execution of a system resource operation of the other control/processing sub-system.

13. In a system resource performing system resource operations in response to requests by the clients, the system resource including a system resource sub-system and a control/processing sub-system including a resource control processor performing system resource operations in response to client requests and controlling operations of the system resource sub-system and including a state machine logging mechanism, a method for logging and restoring the state of execution of system resource operations, comprising the steps of:

during each system resource operation,

extracting state machine information defining a sequence of state machines during an execution of an operation, each state machine representing a current state of execution of a corresponding step in an execution of a system resource operation wherein a state machine is comprised of state information including control and data values representing a state of operation of the control/processing sub-system during the corresponding step in the execution of the system resource operation, and

storing the state machine information, and

upon restoration of operation of the system resource after a failure of system resource operations,

reading the state machine information from the state machine log and restoring the state of execution of a system resource operation, wherein

the control/processing sub-system is a state machine system defined during the execution of an operation by a sequence of state machines defined by states of operation of the state machine system during corresponding steps in the execution of the operation, and wherein

a file transaction operation is represented by at least one sequential state machine wherein each state machine is defined by data and control values residing in the state machine system during existence of state machine of the sequence.

14. The method for logging and restoring the state of execution of system resource operations of claim 13, further comprising the steps of:

during each system resource operation,

storing mirror copies of the state machine information separately from the control/processing sub-system, and

upon restoration of operation of the system resource after a failure of system resource operations,

reading the mirror copies of the state machine information and restoring the state of execution of a system resource operation.

15. In a system resource performing system resource operations in response to system resource requests by the clients, the system resource including a system resource sub-system and first and second control/processing sub-systems, each including a system processor performing system resource operations in response to client requests directed to the first and second control/processing sub-systems and controlling operations of the system resource sub-system, a method for logging and restoring the state of execution of system resource operations, comprising the steps of:

in each control/processor sub-system,

during each system resource operation,

extracting state machine information defining a sequence of state machines during an execution of an operation, each state machine representing a current state of execution of a step in an execution of a system resource operation of the corresponding control/processing sub-system wherein a state machine is comprised of state information including control and data values representing a state of operation of the control/processing sub-system during a corresponding step in the execution of the system resource operation, and

storing the state machine information of the corresponding control/processing sub-system, and

upon restoration of operation of the system resource after a failure of the corresponding control/processing sub-system,

reading the state machine information and restoring the state of execution of a system resource operation of the corresponding control/processing sub-system, wherein

each control/processing sub-system is a state machine system defined during the execution of an operation by a sequence of state machines defined by corresponding states of operation of the state machine system during corresponding steps in the execution of the operation, and wherein

a file transaction operation is represented by a sequence of state machines wherein each state machine is defined by data and control values residing in the state machine system during existence of a state machine of the sequence of state machines.

16. The method for logging and restoring the state of execution of system resource operations of claim 15, further comprising the steps of:

in each control/processing sub-system,

during each system resource operation of the other control/processing sub-system,

receiving and storing mirror copies of the state machine information of the other control/processing sub-system, and

upon restoration of operation of the other control/processing sub-system after a failure of the other control/processing sub-system,

reading the state machine information from the state machine log mirroring mechanism to the other control/processing sub-system and restoring the state of execution of a system resource operation of the other control/processing sub-system.