Enhanced coordinated cluster recovery

• US 7,886,182 B1
• Filed: 04/19/2007
• Issued: 02/08/2011
• Est. Priority Date: 04/19/2007
• Status: Active Grant

• Current Assignee: Network Appliance Incorporated (NetApp, Inc.)
• Original Assignee: Network Appliance Incorporated (NetApp, Inc.)
• Inventors: Ramesh, Deepa, Coatney, Susan M.
• Primary Examiner(s): Bonzo; Bryce P
• Assistant Examiner(s): Ehne; Charles
• Application Number:
  US11/788,661
• Time in Patent Office: 1,391 Days
• Field of Search: 714/3, 714/4, 714/6, 714/11
• US Class Current: 714/3
• CPC Class Codes: G06F 11/2028 eliminating a faulty processor or activating a spare
• CPC Class Codes: G06F 11/2033 switching over of hardware resources
• CPC Class Codes: G06F 11/2038 with a single idle spare processing component
• CPC Class Codes: G06F 11/2046 where the redundant components share persistent storage G06F11/2043 takes precedence

Abstract

An apparatus and a method that prevent a split-brain problem by preventing a cluster partner from accessing and serving data when the cluster partner is taken over by a storage server, while allowing early release of reservations on the cluster partner'"'"'s storage devices before control is given back to the cluster partner.

Claims

1. A method comprising:

operating a storage server in a storage server cluster; and

preventing a cluster partner in the storage server cluster from accessing and serving data from taken-over storage devices associated with the cluster partner when the cluster partner is taken over by the storage server while allowing early release of reservations on the taken-over storage devices associated with the cluster partner before control is given back to the cluster partner,

wherein said preventing comprises sending a takeover state signal representative of the takeover state of the storage server to notify the cluster partner that the cluster partner has been taken over,

wherein the takeover state signal stops the cluster partner from serving the data when the cluster partner does not discover contents of a clustering disk area including clustering information and takeover state information stored on the taken-over storage devices,

wherein the storage server and cluster partner are each physically connected to the taken-over storage devices and the reservations are configured to indicate that the storage server owns the control of the taken-over storage devices until the release of the reservations.

2. The method of claim 1, wherein preventing the cluster partner from booting further comprises:

taking over the control of the taken-over storage devices associated with the cluster partner by the storage server; and

notifying the cluster partner that the cluster partner has been taken over, wherein notifying the cluster partner comprises sending the takeover state signal representative of the takeover state of the storage server over a cluster interconnect between the storage server and the cluster partner.

3. The method of claim 1, further comprising:

releasing the reservations on the taken-over storage devices associated with the cluster partner early; and

giving back the control of the taken-over storage devices to the cluster partner, wherein the reservations are released before said giving back the control to the cluster partner.

4. A method comprising:

initializing a boot process of a storage server in a failover or reboot context;

discovering disk reservations on an array of storage devices, the disk reservations are configured to indicate that a cluster partner owns the control of the array of storage devices until the release of the reservations;

in response to the cluster partner releasing the disk reservations early, searching for contents of a clustering disk area including clustering information and takeover state information;

receiving, at the storage server, takeover data to indicate that the storage server is in a taken-over state; and

determining that the storage server has been taken over by the cluster partner using the contents of the clustering disk area when discovered; and

determining that the storage server has been taken over by the cluster partner using the received takeover data when the contents of the clustering disk area are not discovered, wherein said determining stops the storage server from serving the data from the array of storage devices taken over by the cluster partner while allowing the early release of disk reservations before control is given back to the storage server.

5. The method of claim 4, wherein said receiving takeover data comprises receiving, at the storage server, the takeover data from the cluster partner.

6. The method of claim 4, wherein said receiving takeover data comprises receiving, at the storage server, the takeover data from an external node.

7. The method of claim 6, wherein the external node is at least one of a third storage server coupled to the storage server, or a management console.

8. A system, comprising:

a first storage server physically coupled to communicate with a first array of storage devices and a client; and

a second storage server coupled to the first storage server by a cluster interconnect, wherein the second storage server is physically coupled to communicate with a second array of storage devices and is physically coupled to the first array of storage devices associated with the first storage server, wherein the first storage server is configured to;

initialize a boot process in a failover or reboot context;

discover disk reservations on the first array of storage devices, the disk reservations are configured to indicate that the second server owns the control of the first array of storage devices until the release of the reservations;

in response to the second storage server releasing the disk reservations early, search for contents of a clustering disk area including clustering information and takeover state information;

receive takeover data to indicate that the first storage server is in a taken-over state;

determine that the first storage server is in a taken-over state by the second storage server using the contents of the clustering disk area when discovered; and

determine that the first storage server is in the taken-over state by the second storage server using the received takeover data when the contents of the clustering disk area are not discovered, wherein the takeover data is configured to stop the first storage server from serving the data from the first array storage taken over by the second storage server while allowing the early release of disk reservations before control is given back to the first storage server.

9. The system of claim 8, wherein the first storage server is configured to receive the takeover data from an external device, and wherein the first storage server is configured to determine that the first storage is in the taken-over state using the received data from the external device.

10. The system of claim 9, wherein the external device is at least one of the second storage server, a third storage server coupled to the first storage server, or a management console.

11. A server, comprising:

a processor;

a communication interface through which to communicate with a client of the server;

a second communication interface through which to communicate with taken-over storage devices associated with a cluster partner;

a cluster interconnect adapter to enable the server to communicate with the cluster partner over a cluster interconnect; and

a memory storing instructions which configure the processor to put the server in a takeover state when the cluster partner has failed and to prevent the cluster partner from booting to a point at which the cluster partner serves data, by sending takeover data to the cluster partner to indicate that the server is in the takeover state, and wherein the instructions further configure the processor to allow early release of reservations on the taken-over storage devices associated with the cluster partner before control is given back to the cluster partner, wherein the server and cluster partner are each physically connected to the taken-over-storage devices and the reservations are configured to indicate that the server owns the control of the taken-over storage devices until the release of the reservations, and wherein the takeover data stops the cluster partner from serving the data when the cluster partner does not discover contents of a clustering disk area including clustering information and takeover state information.

12. The server of claim 11, wherein preventing the cluster partner from booting comprises:

taking over the control of the taken-over storage devices associated with the cluster partner by the server; and

notifying the cluster partner that the cluster partner has been taken over, wherein notifying the cluster partner comprises sending the takeover data representative of the takeover state of the storage server over the cluster interconnection between the server and the cluster partner.

13. The method of claim 12, further comprising:

releasing the reservations on the taken-over storage devices associated with the cluster partner early; and

giving back the control of the taken-over storage devices to the cluster partner, wherein the reservations are released before said giving back the control to the cluster partner.

14. A server, comprising:

a processor;

a network adapter coupled to the processor, through which to receive client requests from a client over a network;

a cluster interconnect adapter to enable the server to communicate with a cluster partner; and

a memory storing instructions which configure the processor to respond to receiving takeover data indicating that the server is in a taken-over state to prevent the server from booting to a point at which the server services the client requests in a failover or reboot context, wherein the instructions further configure the processor to initialize a boot process in a failover or reboot context, to discover disk reservations on taken-over resources associated with the server, wherein the reservations are configured to indicate that the server owns the control of the taken-over resources until the release of the reservations, and wherein the instructions further configure the processor to receive the control of the taken-over resources from the cluster partner after the release of reservations, to search for contents of a clustering disk area including clustering information and takeover state information, to determine that the storage server has been taken over by the cluster partner using the contents of the clustering disk area when discovered and using the received takeover data when the contents of the clustering disk area are not discovered, and to continue to boot as part of the boot process in response to receiving the control of the taken-over resources without user intervention at the server.

15. The server of claim 14, wherein receiving data comprises receiving the takeover data from the cluster partner, a third server, or a management console.

16. A method, comprising:

initializing a boot process of a server;

discovering disk reservations on taken-over resources associated with the server, wherein the disk reservations are configured to indicate that the server owns the control of the taken-over resources until the release of the reservations;

receiving, at the server, takeover data to indicate that the server is in a taken-over state by the cluster partner;

searching for contents of a clustering disk area including clustering information and takeover state information;

determining that the server is in the taken-over state by the cluster partner using the contents of the clustering disk area when discovered;

determining that the server is in the taken-over state by the cluster partner using the received takeover data when the contents of the clustering disk area are not discovered;

preventing the server from continuing to boot as part of the boot process in response to determining that the server is in the taken-over state;

receiving the control of the taken-over resources associated with the server from the cluster partner; and

continuing to boot as part of the boot process in response to receiving the control of the taken-over resources without user intervention at the server.

17. The method of claim 16, wherein receiving the takeover data comprises receiving the takeover data from the cluster partner.

18. The method of claim 16, wherein receiving the takeover data comprises receiving the takeover data from a third node.