Integrating control of service during cluster failover

• US 7,694,166 B1
• Filed: 11/30/2005
• Issued: 04/06/2010
• Est. Priority Date: 11/30/2005
• Status: Active Grant

• Current Assignee: Network Appliance Incorporated (NetApp, Inc.)
• Original Assignee: Network Appliance Incorporated (NetApp, Inc.)
• Inventors: Suggs, Darrell G., Fair, Robert L., Kimmel, Jeffrey S., Rowe, Alan L., Sarma, Joydeep Sen
• Primary Examiner(s): Baderman; Scott T
• Assistant Examiner(s): PATEL, KAMINI B
• Application Number:
  US11/292,160
• Time in Patent Office: 1,588 Days
• Field of Search: 714/4
• US Class Current: 714/4.11
• CPC Class Codes: G06F 11/2025 using centralised failover control functionality
• CPC Class Codes: G06F 11/2035 without idle spare hardware
• CPC Class Codes: G06F 11/2046 where the redundant components share persistent storage G06F11/2043 takes precedence

Abstract

A clustered storage server detects an offline condition in another storage server in the cluster and executes a cluster takeover procedure which includes taking control of the storage system associated with the offline storage server, and merging a local control-of-service policy with a control-of-service policy associated with the offline storage server.

Claims

1. A method in a clustered storage server system, comprising:

detecting a failure in a first storage server having a first control-of-service policy to control a first storage system, wherein the first control-of-service policy includes a first plurality of scheduling rules to prioritize file system access requests within the first storage server during normal operation of the first storage server; and

executing a cluster takeover procedure in a second storage server having a second control-of-service policy to control a second storage system, wherein the second control-of-service policy includes a second plurality of scheduling rules to prioritize file system access requests within the second storage server during normal operation of the second storage server, the cluster takeover procedure comprising;

taking control of the first storage system with the second storage server; and

merging the first control-of-service policy with the second control-of-service policy in the second storage server to create a merged control-of-service policy in the second storage server to control the first storage system and the second storage system.

2. The method of claim 1, wherein executing the cluster takeover procedure in the second storage server further comprises transitioning from the second control-of-service policy to the merged control-of-service policy in the second storage server.

3. The method of claim 2, wherein taking control of the first storage system with the second storage server comprises:

activating a path from the second storage server to the first storage system to read control-of-service policy information associated with the first storage system; and

bringing the first storage system online at the second storage server.

4. The method of claim 3, wherein bringing the first storage system online comprises:

mounting the first storage system at the second storage server; and

matching user message priorities of the first storage system to user message priorities of the second storage system.

5. The method of claim 3, wherein transitioning from the second control-of-service policy to the merged control-of-service policy comprises:

lowering user message priorities associated with the second storage system;

lowering system message priorities associated with the second storage system to a lowest priority;

raising system failover message priorities to a highest priority; and

managing user and system message priorities associated with the first storage system, and the user and system message priorities associated with the second storage system, with the merged control-of-service policy.

6. The method of claim 2, further comprising:

detecting a return to service of the first storage server;

executing a cluster giveback procedure in the second storage server comprising;

releasing control of the first storage system to the first storage server under the merged control-of-service policy;

setting user message priorities and system message priorities associated with the second storage system to a lowest priority; and

resuming control of the second storage system under the second control-of-service policy.

7. The method of claim 6, further comprising executing a cluster takeback procedure in the first storage server comprising:

taking control of the first storage system under the merged control-of-service policy;

bringing the first storage system online at the first storage server; and

resuming control of the first storage system under the first control-of-service policy.

8. The method of claim 7, wherein bringing the first storage system online at the first storage server comprises:

mounting the first storage system at the first storage server;

setting user message priorities and system message priorities associated with the first storage system to a highest priority; and

setting the system failover message priorities associated with the first storage server to the highest priority.

9. A system, comprising:

a first storage server coupled to a first storage system and a second storage system, the first storage server configured to control the first storage system according to a first control-of-service policy, wherein the first control-of-service policy includes a first plurality of scheduling rules to prioritize file system access requests within the first storage server during normal operation of the first storage server;

a second storage server coupled with the first storage server, the first storage system and the second storage system, the second storage server configured to

control the second storage system according to a second control-of-service policy, wherein the second control-of-service policy includes a second plurality of scheduling rules to prioritize file system access requests within the second storage server during normal operation of the second storage server,

detect a failure in the first storage server, and

execute a cluster takeover procedure wherein the second storage server is configured to take control of the first storage system, bring the first storage system online, merge the first control-of-service policy with the second control-of-service policy, and control the first storage system and the second storage system according to a merged control-of-service policy.

10. The system of claim 9, wherein the second storage server is further configured to detect a return to service of the first storage server and to execute a cluster giveback procedure, wherein the second storage server is configured to release control of the first storage system to the first storage server under the merged control-of-service policy, and to resume control of the second storage system according to the second control-of-service policy.

11. The system of claim 10, wherein the first storage server is configured to execute a cluster takeback procedure, wherein the first storage server is configured to take control of the first storage system under the merged control-of-service policy, to bring the first storage system online, and to resume control of the first storage system according to the first control-of-service policy.

12. An article of manufacture, comprising a machine-readable storage medium including data that, when accessed by a machine, cause the machine to perform operations comprising a method, the method comprising:

detecting a failure in a first storage server having a first control-of-service policy to control a first storage system, wherein the first control-of-service policy includes a first plurality of scheduling rules to prioritize file system access requests within the first storage server during normal operation of the first storage server; and

executing a cluster takeover procedure in a second storage server having a second control-of-service policy to control a second storage system, wherein the second control-of-service policy includes a second plurality of scheduling rules to prioritize file system access requests within the second storage server during normal operation of the second storage server, the cluster takeover procedure comprising;

taking control of the first storage system with the second storage server; and

merging the first control-of-service policy with the second control-of-service policy in the second storage server to create a merged control-of-service policy in the second storage server to control the first storage system and the second storage system.

13. The article of manufacture of claim 12, wherein executing the cluster takeover procedure in the second storage server further comprises transitioning from the second control-of-service policy to the merged control-of-service policy in the second storage server.

14. The article of manufacture of claim 13, wherein taking control of the first storage system with the second storage server comprises:

activating a path from the second storage server to the first storage system to read control-of-service policy information associated with the first storage system; and

bringing the first storage system online at the second storage server.

15. The article of manufacture of claim 14, wherein bringing the first storage system online comprises:

mounting the first storage system at the second storage server; and

matching user message priorities of the first storage system to user message priorities of the second storage system.

16. The article of manufacture of claim 14, wherein transitioning from the second control-of-service policy to the merged control-of-service policy comprises:

lowering user message priorities associated with the second storage system;

lowering system message priorities associated with the second storage system to a lowest priority;

raising system failover message priorities to a highest priority; and

managing user and system message priorities associated with the first storage system, and the user and system message priorities associated with the second storage system, with the merged control-of-service policy.

17. The article of manufacture of claim 13, the method further comprising:

detecting a return to service of the first storage server; and

executing a cluster giveback procedure in the second storage server comprising;

releasing control of the first storage system to the first storage server under the merged control-of-service policy;

setting user message priorities and system message priorities associated with the second storage system to a lowest priority; and

resuming control of the second storage system under the second control-of-service policy.

18. The article of manufacture of claim 17, the method further comprising executing a cluster takeback procedure in the first storage server comprising:

taking control of the first storage system under the merged control-of-service policy;

bringing the first storage system online at the first storage server; and

resuming control of the first storage system under the first control-of-service policy.

19. The article of manufacture of claim 18, wherein bringing the first storage system online at the first storage server comprises:

mounting the first storage system at the first storage server;

setting user message priorities and system message priorities associated with the first storage system to a highest priority; and

setting the system failover message priorities associated with the first storage server to the highest priority.

20. A storage server, comprising:

a cluster interface to communicate with another storage server having a first control-of-service policy to control a first storage system, wherein the first control-of-service policy includes a first plurality of scheduling rules to prioritize file system access requests within the first storage server during normal operation of the first storage server;

one or more storage adapters to communicate with the first storage system and a second storage system;

a processor coupled with the cluster interface and the one or more storage adapters,

the processor to implement a storage operating system having a second control-of-service policy to control the second storage system, wherein the second control-of-service policy includes a second plurality of scheduling rules to prioritize file system access requests within the second storage server during normal operation of the second storage server,

the processor further to detect a failure in the other storage server, and to execute a takeover procedure to take control of the first storage system, merge the first control-of-service policy with the second control-of-service policy, and control the first storage system and the second storage system with a merged control-of-service policy.

21. The storage server of claim 20, the processor further to transition the storage server from the second control-of-service policy to the merged control-of-service policy.

22. The storage server of claim 21, the processor further to bring the first storage system online, wherein the processor is configured to mount logical volumes of the first storage system and match user message priorities associated with the first storage system to user message priorities associated with the second storage system.

23. The storage server of claim 21, wherein to transition from the second control-of-service policy to the merged control-of-service policy, the processor further to:

lower user message priorities associated with the second storage system;

lower system message priorities associated with the second storage system to a lowest priority;

raise system failover message priorities to a highest priority; and

manage user and system message priorities associated with the first storage system, and the user and system message priorities associated with the second storage system, with the merged control-of-service policy.

24. The storage server of claim 21, the processor further to:

detect a return to service of the first storage server;

execute a cluster giveback procedure, wherein the processor is configured to;

release control of the first storage system to the first storage server under the merged control-of-service policy;

set user message priorities and system message priorities associated with the second storage system to a lowest priority; and

resume control of the second storage system under the second control-of-service policy.