Coordinating persistent status information with multiple file servers

• US 6,829,720 B2
• Filed: 12/16/2002
• Issued: 12/07/2004
• Est. Priority Date: 08/25/1998
• Status: Expired due to Fees

• Current Assignee: NetApp, Inc.
• Original Assignee: Network Appliance Incorporated (NetApp, Inc.)
• Inventors: Schoenthal, Scott, Rowe, Alan, Kleiman, Steven R.
• Primary Examiner(s): Bonzo, Bryce P.
• Application Number:
  US10/321,832
• Publication Number:
  US 20030149812A1
• Time in Patent Office: 722 Days
• Field of Search: 714/6, 714/4, 714/7, 714/8, 714/15, 714/18, 709/213, 709/214, 709/215, 709/216
• US Class Current: 714/4.4
• CPC Class Codes: G06F 11/142 Reconfiguring to eliminate the error group management mechanisms in a peer-to-peer network H04L67/1044
• CPC Class Codes: G06F 11/2007 using redundant communication media
• CPC Class Codes: G06F 11/2033 switching over of hardware resources
• CPC Class Codes: G06F 11/2041 with more than one idle spare processing component
• CPC Class Codes: G06F 11/2048 where the redundant components share neither address space nor persistent storage
• CPC Class Codes: G06F 11/2097 maintaining the standby controller/processing unit updated initialisation or re-synchronisation thereof G06F11/1658 and subgroups

Abstract

The invention provides a storage system, and a method for operating a storage system, that provides for relatively rapid and reliable takeover among a plurality of independent file servers. Each file server maintains a reliable communication path to the others. Each file server maintains its own state in reliable memory. Each file server regularly confirms the state of the other file servers. Each file server labels messages on the redundant communication paths, so as to allow other file servers to combine the redundant communication paths into a single ordered stream of messages. Each file server maintains its own state in its persistent memory and compares that state with the ordered stream of messages, so as to determine whether other file servers have progressed beyond the file server'"'"'s own last known state. Each file server uses the shared resources (such as magnetic disks) themselves as part of the redundant communication paths, so as to prevent mutual attempts at takeover of resources when each file server believes the other to have failed. Each file server provides a status report to the others when recovering from an error, so as to prevent the possibility of multiple file servers each repeatedly failing and attempting to seize the resources of the others.

Claims

1. A file server, including:

an interface to a set of mass storage devices and to at least one network; and

a processor and controller disposed to access said mass storage devices, to communicate messages with at least a second file server that has access to said mass storage devices, and to process state information about said server and said second file server;

wherein said messages are used to communicate said state information to and from said second file server, and wherein said messages are sent over plural different communication paths including at least part of said mass storage devices and said network.

2. A file server as in claim 1, wherein said part of said mass storage devices that are included in said communication paths further comprises one or more mailboxes stored on said mass storage devices.

3. A file server as in claim 1, wherein said plural different communication paths include at least one other network.

4. A file server as in claim 1, wherein one of the servers can take over control of the mass storage devices from the other server.

5. A file server as in claim 4, wherein take over occurs if messages from the other server timeout or if messages from the other server indicate that the other server has changed state.

6. A file server as in claim 5, wherein different timeouts are used for the different communication paths.

7. A file server as in claim 4, wherein said messages are used to prevent both servers from concurrently attempting to take over control of the mass storage devices.

8. A method of controlling a file server, comprising the steps of:

accessing a set of mass storage devices;

communicating messages with at least a second file server that has access to said mass storage devices; and

processing state information about said server and said second file server;

wherein said messages are used to communicate said state information to and from said second file server, and wherein said messages are sent over plural different communication paths including at least part of said mass storage devices and a network.

9. A method as in claim 8, wherein said part of said mass storage devices that are included in said communication paths further comprises one or more mailboxes stored on said mass storage devices.

10. A method as in claim 8, wherein said plural different communication paths include at least one other network.

11. A method as in claim 8, wherein one of the servers can take over control of the mass storage devices from the other server.

12. A method as in claim 11, wherein take over occurs if messages from the other server timeout or if messages from the other server indicate that the other server has changed state.

13. A method as in claim 12, wherein different timeouts are used for the different communication paths.

14. A method as in claim 11, wherein said messages are used to prevent both servers from concurrently attempting to take over control of the mass storage devices.

15. A memory storing information including instructions, the instructions executable by a processor to control a file server, the instructions comprising the steps of:

accessing a set of mass storage devices;

communicating messages with at least a second file server that has access to said mass storage devices; and

processing state information about said server and said second file server;

wherein said messages are used to communicate said state information to and from said second file server, and wherein said messages are sent over plural different communication paths including at least part of said mass storage devices and a network.

16. A memory as in claim 15, wherein said part of said mass storage devices that are included in said communication paths further comprises one or more mailboxes stored on said mass storage devices.

17. A memory as in claim 15, wherein said plural different communication paths include at least one other network.

18. A memory as in claim 15, wherein one of the servers can take over control of the mass storage devices from the other server.

19. A memory as in claim 18, wherein take over occurs if messages from the other server timeout or if messages from the other server indicate that the other server has changed state.

20. A memory as in claim 19, wherein different timeouts are used for the different communication paths.

21. A memory as in claim 18, wherein said messages are used to prevent both servers from concurrently attempting to take over control of the mass storage devices.

22. A storage system comprising:

a plurality of mass storage devices comprising a first subset of mass storage devices and a second subset of mass storage devices;

a first file server comprising a first interface to the plurality of mass storage devices, and a first processor and first controller being capable of accessing the plurality of mass storage devices, the first file server being configured to be a primary controller for the first subset of mass storage devices; and

a second file server comprising a second interface to the plurality of mass storage devices, and a second processor and second controller being capable of accessing the plurality of mass storage devices, the second file server being configured to be a primary controller for the second subset of mass storage devices;

wherein

the first processor and controller and the second processor and controller are further configured to communicate messages with state information of the first and second file servers between the first and second file servers over plural different communication paths, the plural different communication paths comprising at least some storage within said plurality of mass storage devices; and

the first file server is capable of accessing the second subset of mass storage devices when the second file server fails, and the second file server is capable of accessing the first subset of mass storage devices when the first file server fails.

23. A storage system as in claim 22, wherein the first file server is capable of taking over control of the second subset of mass storage devices when the first file server detects failure of the second file server, and the second file server is capable of taking over control of the first subset of mass storage devices when the second file server detects failure of the first file server.

24. A storage system as in claim 23, wherein messages sent by the first file server comprise a state indicator for the first file server, the state indicator being selected from normal, takeover, no-takeover, and disable indications, wherein the normal indication signifies normal operation, the takeover indication signifies that the first file server has taken over control of the second subset of mass storage devices, the no-takeover indication signifies that the second file server should not take over control of the first subset of mass storage devices, and the disable indication signifies that take over of the mass storage devices is disabled for the first and the second file servers.

25. A storage system as in claim 23, wherein messages sent by the first file server comprise a monotonically increasing generation number associated with a current instantiation of the first file server.

26. A storage system as in claim 23, wherein messages sent by the first file server comprise a monotonically increasing message sequence number.

27. A storage system as in claim 23, wherein the first file server further comprises a first persistent memory wherein the second file server maintains state information of the first file server, and the second file server further comprises a second persistent memory wherein the second file server maintains state information of the second file server.