System and method for coordinated bringup of a storage appliance in a cluster configuration

• US 8,028,054 B1
• Filed: 10/02/2007
• Issued: 09/27/2011
• Est. Priority Date: 01/29/2004
• Status: Active Grant

• Current Assignee: NetApp, Inc.
• Original Assignee: NetApp, Inc.
• Inventors: Sun, Ye, Coatney, Susan M., Watanabe, Steven S., Rowe, Alan L.
• Primary Examiner(s): Jaroenchonwanit; Bunjob
• Application Number:
  US11/866,011
• Time in Patent Office: 1,456 Days
• Field of Search: 709/212
• US Class Current: 709/223
• CPC Class Codes: G06F 11/2028 eliminating a faulty processor or activating a spare

Abstract

A system and method for coordinated bringup of a storage appliance in a storage appliance cluster. The repaired storage appliance, during its initialization, sets a variety of state values in a predetermined memory location comprising a state data structure, which is detected by a remote direct memory access read operation by the surviving storage appliance. By the use of the RDMA operations, the repaired storage appliance and surviving storage appliance coordinate the bringup and giveback of data servicing functionality.

Claims

1. A system, comprising:

a first storage system;

a second storage system coupled to the first storage system and configured to detect a failure of the first storage system, upon detection of the failure of the first storage system the second storage system placing reservations on data storage devices used by the first storage system, and the second storage system to begin servicing data access requests directed to the first storage system;

after repair of the first storage system, the first storage system to begin a boot operation, the first storage system to detect the reservations placed by the second storage system on the data storage devices, and in response to detecting the reservations halting the boot operation and writing to a memory a first state indicating that the first storage system is booting;

the second storage system reading the first state, and in response to reading the first state releasing the reservations and writing a second state into the memory;

in response to the second state written into the memory, the first storage system initializing the data storage devices, and writing a first message into the memory;

in response to the first message, the second storage system to no longer service the data access requests directed to the first storage system, the second storage system to release resources including the data storage devices to the first storage system, and the second storage system to write a second message into the memory; and

in response to the second message, the first storage system completes the boot operation and begins processing the data access requests directed to the first storage system.

2. The system as in claim 1, further comprising:

the resources released by the second storage system include network addresses utilized in servicing the data access requests directed to the repaired first storage system.

3. The system as in claim 1, further comprising:

the data storage devices include disks.

4. The system as in claim 1, further comprising:

the data storage devices include computer readable media.

5. The system as in claim 1, further comprising:

the data storage devices include memories.

6. The system as in claim 1, further comprising:

the data storage devices include tapes.

7. The system of claim 1, further comprising:

the second storage system to execute a giveback operation, the giveback operation to release one or more of the resources taken over in response to the failure of the first storage system.

8. The system of claim 7, further comprising:

the one or more resources are one or more of the network addresses.

9. The system of claim 1, further comprising:

the second message is a mailbox state that indicates that the second storage system is no longer servicing the data access requests directed to the first storage system.

10. The system of claim 9, further comprising:

the first storage system is further configured to detect the second message, complete initialization, and to process the data access requests directed to the first storage system.

11. The system of claim 1, further comprising:

the second storage system performs remote direct memory address (RDMA) read operations to read the memory within the first storage system.

12. The system as in claim 1, further comprising:

the resources released by the second storage system include the network addresses utilized in servicing the data access requests directed to the repaired first storage system.

13. A method for operating a first storage system operatively coupled to a second storage system, comprising:

detecting a failure of the first storage system by the second storage system;

placing reservations by the second storage system on data storage devices used by the first storage system, and the second storage system servicing data access requests directed to the first storage system;

after repair of the first storage system, beginning a boot operation by the first storage system, the first storage system detecting the reservations placed by the second storage system on the data storage devices, and in response to detecting the reservations halting the boot operation and writing to a memory a first state indicting that the first storage system is booting;

reading by the second storage system the first state, and in response to reading the first state, the second storage system releasing the reservations and writing a second state into the memory;

in response to the second state written into the memory, initializing by the first storage system the data storage devices, and writing a first message into the memory;

in response to the first message, the second storage system no longer servicing the data access requests directed to the first storage system, and the second storage system releasing resources including the data storage devices to the first storage system, and the second storage system writing a second message into the memory; and

in response to the second message, the first storage system completing the boot operation, and beginning processing the data access requests directed to the first storage system.

14. The method as in claim 13, further comprising:

including network addresses utilized in servicing the data access requests directed to the repaired first storage system in the resources released by the second storage system.

15. The method as in claim 13, further comprising:

using disks as the data storage devices.

16. The method as in claim 13, further comprising:

using computer readable media as the data storage devices.

17. The method as in claim 13, further comprising:

using memories as the data storage devices.

18. The method as in claim 13, further comprising:

using tapes as the data storage devices.

19. The method of claim 13, further comprising:

executing a giveback operation by the second storage system, the giveback operation to release one or more of the resources taken over in response to a failure of the first storage system.

20. The method of claim 19, further comprising:

including one or more network addresses as the one or more resources.

21. The method of claim 13, further comprising:

indicating that the second storage system is no longer servicing the data access requests directed to the first storage system by the second message.

22. The method of claim 13, further comprising:

configuring the first storage system to detect the second message, complete initialization, and to process the data access requests directed to the first storage system.

23. The method of claim 13, further comprising:

performing by the second storage system remote direct memory address (RDMA) read operations to read the memory within the first storage system.

24. The method as in claim 13, further comprising:

including TCP/IP protocols utilized in servicing data access requests directed to the repaired first storage system in the resources released by the second storage system.

25. A computer readable non-transitory storage media, comprising:

said computer readable storage media containing program instructions for execution on a processor for a method of operating a first storage system operationally coupled to a second storage system, the program instructions for,

detecting a failure of the first storage system by the second storage system;

placing reservations by the second storage system on data storage devices used by the first storage system, and the second storage system servicing data access requests directed to the first storage system;

after repair of the first storage system, beginning a boot operation by the first storage system, the first storage system detecting the reservations placed by the second storage system on the data storage devices, and in response to detecting the reservations halting the boot operation and writing to a memory a first state indicting that the first storage system is booting;

reading by the second storage system the first state, and in response to reading the first state, the second storage system releasing the reservations and writing a second state into the memory;

in response to the second state written into the memory, initializing by the first storage system the data storage devices, and writing a first message into memory;

in response to the first message, the second storage system no longer servicing the data access requests directed to the first storage system, and the second storage system releasing resources including the data storage devices to the first storage system, and the second storage system writing a second message into the memory; and

in response to the second message, the first storage system completing the boot operation, and beginning processing the data access requests directed to the first storage system.