Apparatus and method for deterministically performing active-active failover of redundant servers in response to a heartbeat link failure

• US 20050010838A1
• Filed: 04/23/2004
• Published: 01/13/2005
• Est. Priority Date: 04/23/2003
• Status: Active Grant

• Current Assignee: Dot Hill Systems Corporation (Seagate Technology Holdings Plc)
• Original Assignee: Dot Hill Systems Corporation (Seagate Technology Holdings Plc)
• Inventors: Davies, Ian Robert, Pecone, Victor Key, Kalwitz, George Alexander
• Granted Patent: US 7,380,163 B2
• Time in Patent Office: Days
• US Class Current: 714/100
• CPC Class Codes: G06F 11/1456 Hardware arrangements for backup
• CPC Class Codes: G06F 11/1464 for networked environments
• CPC Class Codes: G06F 11/2005 using redundant communication controllers
• CPC Class Codes: G06F 11/2007 using redundant communication media
• CPC Class Codes: G06F 11/201 between storage system components
• CPC Class Codes: G06F 11/2028 eliminating a faulty processor or activating a spare
• CPC Class Codes: G06F 11/2035 without idle spare hardware
• CPC Class Codes: G06F 11/2046 where the redundant components share persistent storage G06F11/2043 takes precedence
• CPC Class Codes: G06F 11/2051 in regular structures
• CPC Class Codes: G06F 11/2092 Techniques of failing over between control units
• CPC Class Codes: G06F 3/0601 Interfaces specially adapted for storage systems
• CPC Class Codes: G06F 3/0673 Single storage device

Abstract

An apparatus is disclosed for deterministically performing active-active failover of redundant servers in response to a failure of a link on which each server provides a heartbeat to the other server. Each of the servers is configured to take over the identity of the other server on a common network in response to detecting a failure of the other server'"'"'s link heartbeat. Each server provides a status indicator to a storage controller indicating whether the other server'"'"'s link heartbeat stopped. The storage controller determines the link has failed if both of the status indicators indicate the other server'"'"'s heartbeat stopped, and responsively kills one of the servers. The storage controller also receives a heartbeat directly from each server. If only one direct heartbeat stops when the status indicators indicate the link heartbeats stopped, then the storage controller detects one server has failed and inactivates the failed server.

Claims

1. An apparatus for deterministically performing active-active failover of redundant servers in response to a failure of a link coupling the servers on which each server provides a heartbeat to the other server, wherein each of the servers is configured to take over the identity of the other server on a common network in response to detecting a failure of the other server'"'"'s heartbeat on the link, the apparatus comprising:

a status indicator, provided by each of the servers to indicate whether the other server'"'"'s heartbeat has failed; and

a storage controller, coupled to receive each of said status indicators from the servers, wherein said storage controller determines the link has failed if each of said status indicators indicates the other server'"'"'s heartbeat has failed, wherein said storage controller inactivates one of the servers in response to determining the link has failed.

2. The apparatus of claim 1, wherein said storage controller inactivates said one of the servers in response to each of said status indicators indicating the other server'"'"'s heartbeat has failed, prior to said non-inactivated server taking over the identity of said inactivated server.

3. The apparatus of claim 1, further comprising:

a second status indicator, provided by said storage controller to each of the servers, for indicating whether said storage controller has inactivated one of the servers, wherein said non-inactivated server waits to take over the identity of the other server on the network until said second status indicator indicates said storage controller has inactivated one of the servers.

4. The apparatus of claim 1, wherein said non-inactivated server waits to take over the identity of the other server on the network until a predetermined time period has expired after said non-inactivated server detects the failure of the other server'"'"'s heartbeat on the link.

5. The apparatus of claim 1, wherein said storage controller inactivating one of the servers comprises preventing said one of the servers from participating on the network.

6. The apparatus of claim 1, wherein said storage controller inactivating one of the servers comprises preventing said one of the servers from responding to packets on the network.

7. The apparatus of claim 1, wherein said storage controller inactivating one of the servers comprises preventing said one of the servers from transmitting packets on the network.

8. The apparatus of claim 1, wherein said storage controller inactivating one of the servers comprises resetting said one of the servers.

9. The apparatus of claim 1, wherein said storage controller inactivating one of the servers comprises resetting an I/O port of said one of the servers for communicating on the network.

10. The apparatus of claim 1, wherein said storage controller inactivating one of the servers comprises causing power to be removed from said one of the servers.

11. The apparatus of claim 1, wherein said storage controller inactivating one of the servers comprises inactivating said one of the servers independent of an operational state of said one of the servers.

12. The apparatus of claim 1, further comprising:

a second status indicator, provided by each of the servers to said storage controller, for indicating a health of the server providing said second status indicator, wherein if said second status indicators indicate one of the servers is less healthy than the other server, said storage controller inactivates said less healthy one of the servers in response determining the link has failed.

13. The apparatus of claim 1, further comprising:

a second status indicator, provided by each of the servers to said storage controller, for indicating whether the server providing said status indicator has failed, wherein if one of said second status indicators indicates the server providing said status indicator has failed, said storage controller inactivates said failed server providing said status indicator.

14. The apparatus of claim 1, further comprising:

an inactivation control, generated by said storage controller to each of the servers, for inactivating the respective server.

15. The apparatus of claim 14, further comprising:

a backplane, configured to couplably receive each of the servers and said storage controller, comprising at least a portion of each of said inactivation controls.

16. The apparatus of claim 15, wherein said backplane comprises at least a portion of each of said status indicators.

17. The apparatus of claim 15, further comprising:

a chassis, for enclosing the servers, said storage controller, and said backplane.

18. The apparatus of claim 17, wherein the link is enclosed in said chassis.

19. The apparatus of claim 18, wherein said backplane comprises at least a portion of the link.

20. The apparatus of claim 17, wherein the link is external to said chassis.

21. The apparatus of claim 17, wherein said chassis comprises a chassis for mounting in a 19 inch wide rack.

22. The apparatus of claim 1, wherein said storage controller and the servers comprise a plurality of hot-pluggable field replaceable units (FRUs).

23. The apparatus of claim 1, further comprising:

a second storage controller, coupled to receive said status indicator from each of the servers, configured to determine whether said first storage controller has failed, and if so, to inactivate one of the servers in response to determining from said status indicators that the link has failed.

24. The apparatus of claim 23, wherein said first and second storage controllers and the servers comprise a plurality of hot-pluggable field replaceable units (FRUs).

25. The apparatus of claim 24, wherein any one of said FRUs may be hot-replaced without suffering loss of data availability to the network of a plurality of storage devices controlled by said first and second storage controllers.

26. The apparatus of claim 24, wherein said plurality of FRUs comprise blade modules.

27. The apparatus of claim 1, wherein said storage controller comprises a redundant array of inexpensive disks (RAID) controller.

28. The apparatus of claim 1, wherein the servers are configured to execute storage application software.

29. A method for deterministically performing active-active failover of first and second redundant servers, the method comprising:

indicating, by the first server, a first heartbeat previously received on a link from a second server has stopped;

indicating, by the second server, a second heartbeat previously received on the link from the first server has stopped;

determining, by a storage controller, the link has failed in response to said first and second servers indicating the first and second heartbeats on the link have stopped; and

inactivating, by the storage controller, one of the servers in response to said determining.

30. The method of claim 29, further comprising:

determining, by a second storage controller, the first storage controller has failed, prior to said determining the link has failed;

determining, by the second storage controller, the link has failed in response to said first and second servers indicating the first and second heartbeats on the link have stopped; and

inactivating, by the second storage controller, one of the servers in response to said determining.

31. The method of claim 29, further comprising:

receiving, by the storage controller, from each of the first and second servers an indicator indicating a health of said server, prior to said inactivating; and

inactivating, by the storage controller, one of the servers having a worst health in response to said receiving the health indicator.

32. A network storage appliance, comprising:

a storage controller; and

redundant servers, coupled to said storage controller, configured to perform active-active failover on a network to which each is coupled, each configured to generate;

a first heartbeat, to the other server on a link coupling said servers;

a second heartbeat, to said storage controller; and

a status, to said storage controller, of said heartbeat received from the other server;

wherein said storage controller is configured to distinguish between a first situation in which one of said redundant servers has failed and a second situation in which said link has failed based on said first heartbeats, said second heartbeats, and said statuses.

33. The network storage appliance of claim 32, wherein said storage controller determines said first condition has occurred if said status from both of said servers stops and said second heartbeat from only one of said servers stops.

34. The network storage appliance of claim 32, wherein said storage controller determines said second condition has occurred if said status from both of said servers stops and said second heartbeat from neither of said servers stops.

35. The network storage appliance of claim 32, further comprising:

an inactivation control, generated by said storage controller, to inactivate one of said servers in either of said first and second situations, thereby enabling the other redundant server to deterministically perform failover for said server inactivated by said storage controller.

36. The network storage appliance of claim 35, wherein in said second situation said storage controller is configured to select one of said redundant servers, and to inactivate said selected one of said redundant servers.

37. The network storage appliance of claim 35, wherein in said first situation said storage controller is configured to inactivate said failed one of said redundant servers.

38. The network storage appliance of claim 35, wherein each of said redundant servers communicates on said network using at least one respective unique ID.

39. The network storage appliance of claim 38, wherein said redundant servers performing active-active failover on said network comprises said non-inactivated server taking over said at least one respective unique ID of said inactivated server.

40. The network storage appliance of claim 32, wherein said link comprises a FibreChannel link.

41. The network storage appliance of claim 32, wherein said link comprises an Infiniband link.

42. The network storage appliance of claim 32, wherein said link comprises an Ethernet link.

43. The network storage appliance of claim 32, further comprising:

a chassis, enclosing said storage controller and said redundant servers, wherein said link comprises a link internal to said chassis.

44. The network storage appliance of claim 43, further comprising:

a backplane, enclosed in said chassis, wherein said storage controller and said redundant servers comprise a plurality of blade modules for plugging into said backplane, wherein said link is substantially comprised in said backplane.