Coordinating persistent status information with multiple file servers

• US 6,119,244 A
• Filed: 08/25/1998
• Issued: 09/12/2000
• Est. Priority Date: 08/25/1998
• Status: Expired due to Term

• Current Assignee: NetApp, Inc.
• Original Assignee: Network Appliance Incorporated (NetApp, Inc.)
• Inventors: Kleiman, Steven R., Schoenthal, Scott, Rowe, Alan
• Primary Examiner(s): Beausoliel, Jr., Robert W.
• Assistant Examiner(s): BONZO, BRYCE P
• Application Number:
  US09/139,257
• Time in Patent Office: 749 Days
• Field of Search: 714/4, 714/5, 714/6, 714/7, 714/8, 714/15, 714/18, 714/10, 714/11, 714/12, 714/13, 709/213, 709/214, 709/215, 709/216
• US Class Current: 714/4.4
• CPC Class Codes: G06F 11/142 Reconfiguring to eliminate the error group management mechanisms in a peer-to-peer network H04L67/1044
• CPC Class Codes: G06F 11/2007 using redundant communication media
• CPC Class Codes: G06F 11/2033 switching over of hardware resources
• CPC Class Codes: G06F 11/2041 with more than one idle spare processing component
• CPC Class Codes: G06F 11/2048 where the redundant components share neither address space nor persistent storage
• CPC Class Codes: G06F 11/2097 maintaining the standby controller/processing unit updated initialisation or re-synchronisation thereof G06F11/1658 and subgroups

Abstract

The invention provides a storage system, and a method for operating a storage system, that provides for relatively rapid and reliable takeover among a plurality of independent file servers. Each file server maintains a reliable communication path to the others. Each file server maintains its own state in reliable memory. Each file server regularly confirms the state of the other file servers. Each file server labels messages on the redundant communication paths, so as to allow other file servers to combine the redundant communication paths into a single ordered stream of messages. Each file server maintains its own state in its persistent memory and compares that state with the ordered stream of messages, so as to determine whether other file servers have progressed beyond the file server'"'"'s own last known state. Each file server uses the shared resources (such as magnetic disks) themselves as part of the redundant communication paths, so as to prevent mutual attempts at takeover of resources when each file server believes the other to have failed. Each file server provides a status report to the others when recovering from an error, so as to prevent the possibility of multiple file servers each repeatedly failing and attempting to seize the resources of the others.

Claims

1. A file server including

a set of storage devices capable of being shared with a second file server;

a controller disposed for coupling to said shared set of storage devices;

a transceiver disposed for coupling to a communication path and for communicating messages using said communication path, said communication path using said shared set of storage devices to communicate said messages;

a takeover monitor coupled to at least part of said shared set of storage devices, and responsive to said communication path and said shared set of storage devices.

2. A file server as in claim 1, including persistent memory storing state information about said file server, said takeover monitor being responsive to said persistent memory.

3. Apparatus including

a shared resource;

a pair of servers each coupled to said shared resource and each disposed for managing at least part of said shared resource;

a communication path disposed for coupling a sequence of messages between said pair, said communication path disposed for using s aid shared resource for coupling said sequence of messages;

each one of said pair being disposed for takeover of at least part of said shared resource in response to said communication path;

whereby said communication path prevents both of said pair from concurrently performing said takeover.

4. Apparatus as in claim 3, wherein

at least one said server includes a file server;

said shared resource includes a storage medium; and

said communication path includes a designated location on said storage medium.

5. Apparatus as in claim 3, wherein

each one of said pair includes persistent memory;

said persistent memory being disposed for storing state information about said pair; and

each one of said pair being disposed for takeover in response to said persistent memory.

6. Apparatus as in claim 3, wherein

each said server is disposed for transmitting a message including recovery information relating to a status of said server on recovery from a service interruption;

and

each said server is disposed so that giveback of at least part of said shared resource is responsive to said recovery information.

7. Apparatus as in claim 3, wherein

each said server is disposed for transmitting a message including recovery information relating to a status of said server on recovery from a service interruption;

and

each said server is disposed so that said takeover is responsive to said recovery information.

8. Apparatus as in claim 3, wherein

said pair includes a first server and a second server;

said first server determines a state for itself and for said second server in response to said communication path;

said second server determines a state for itself and for said first server in response to said communication path;

whereby said first server and said second server concurrently each determine state for each other, such that it does not occur that each of said first server and said second server both consider the other to be inoperative.

9. Apparatus as in claim 3, wherein

said shared resource includes a plurality of storage devices;

and

said communication path includes at least part of said storage devices.

10. Apparatus as in claim 3, wherein

said communication path includes a plurality of independent communication paths between said pair;

and

each message in said sequence includes a generation number, said generation number being responsive to a service interruption and a persistent memory for a sender of said message.

11. Apparatus as in claim 3, wherein

said communication path includes a plurality of independent communication paths between said pair;

and

said first server is disposed for determining a state for itself and for said second server in response to a state of said shared resource and in response to a state of a persistent memory at said first server.

12. Apparatus as in claim 3, wherein

said communication path includes a plurality of independent communication paths between said pair;

and

said plurality of independent communication paths includes at least two of the group;

a packet network, a shared storage element, a system area network.

13. Apparatus as in claim 3, wherein

said communication path is disposed for transmitting at least one message from a first said server to a second said server;

said message indicating that said first server is attempting said takeover;

receipt of said message being responsive to a state of said shared resource.

14. Apparatus as in claim 13, wherein said second server is disposed for altering its state in response to said message, in said altered state refraining from writing to said shared resource.

15. A method for operating a file server, said method including steps for controlling a subset of a set of shared storage devices;

receiving and transmitting messages with a second file server, said steps for receiving and transmitting using a communication path including said shared storage devices;

monitoring said communicating path and said shared storage devices;

storing state information about said file server in a persistent memory; and

performing a takeover operation of said shared resource in response to said steps for monitoring and a state of said persistent memory.

16. A method including steps for

managing at a first server at least a part of a shared resource;

receiving and transmitting a sequence of messages between said first server to a second server, using said shared resource;

performing a takeover operation at a first server of at least part of said shared resource in response to said sequence of messages;

whereby said steps for receiving and transmitting prevent both of said first server and said second server from concurrently performing said takeover operation.

17. A method as in claim 16, including steps for

determining, at said first server, a state for itself and for said second server in response to a communication path;

determining, at said second server, a state for itself and for said first server in response to said communication path;

whereby said first server and said second server concurrently each determine state for each other, such that it does not occur that each of said first server and said second server both consider the other to be inoperative.

18. A method as in claim 16, including steps for storing state information about said first server in a persistent memory, wherein said first server determines a state for itself in response to a state of said persistent memory.

19. A method as in claim 16, including steps for

transmitting, from said first server, recovery information relating to a status of said first server on recovery from a service interruption;

and

performing a giveback operation of at least part of said shared resource is responsive to said recovery information.

20. A method as in claim 16, including steps for

transmitting, from said first server, recovery information relating to a status of said server on recovery from a service interruption;

wherein said steps for performing said takeover operation are responsive to said recovery information.

21. A method as in claim 16, wherein

said shared resource includes a plurality of storage devices;

and

a communication path includes at least part of said storage devices;

whereby loss of access to said part of said storage devices breaks said communication path.

22. A method as in claim 16, including steps for

transmitting at least one message from a first said server to a second said server, said message indicating that said first server is attempting said takeover;

altering a state of said second server in response to said message; and

in said altered state refraining from writing to said shared resource.

23. A method as in claim 16, wherein a communication path includes a plurality of independent communication paths between said pair;

and including steps for

numbering said sequence of messages;

determining, at each recipient, a unified order for messages delivered using different ones of said plurality of independent communication paths; and

determining, at said first server, a state for itself and for said second server in response to a state of said shared resource and in response to a state of a persistent memory at said first server.

24. A method as in claim 16, wherein a communication path includes a plurality of independent communication paths between said pair;

and including steps for

numbering said sequence of messages;

determining, at each recipient, a unified order for messages delivered using different ones of said plurality of independent communication paths;

transmitting substantially each message in said sequence on at least two of said plurality of independent communication paths, whereby there is no single point of failure for communication between said pair.

25. A method as in claim 16, wherein a communication path includes a plurality of independent communication paths between said pair;

and including steps for

numbering said sequence of messages;

determining, at each recipient, a unified order for messages delivered using different ones of said plurality of independent communication paths;

wherein said plurality of independent communication paths includes at least two of the group;

a packet network, a shared storage element, a system area network.

26. A method as in claim 16, wherein a communication path includes a plurality of independent communication paths between said pair;

and including steps for

numbering said sequence of messages;

determining, at each recipient, a unified order for messages delivered using different ones of said plurality of independent communication paths;

wherein said steps for numbering include (a) determining a generation number in response to a service interruption and a persistent memory for a sender of said message, and (b) providing said generation number in substantially each message in said sequence.