Multi-tier mechanism to achieve high availability in a multi-controller system

• US 10,203,890 B1
• Filed: 09/20/2016
• Issued: 02/12/2019
• Est. Priority Date: 09/20/2016
• Status: Active Grant

• Current Assignee: Tintri Incorporated
• Original Assignee: Tintri, Inc.
• Inventors: Lathia, Chirag, Vutukuru, Amar
• Primary Examiner(s): Thai, Tuan V
• Assistant Examiner(s): Matin, Tasnima
• Application Number:
  US15/271,156
• Time in Patent Office: 875 Days
• CPC Class Codes: G06F 11/0757 by exceeding a time limit, i.e. time-out, e.g. watchdogs
• CPC Class Codes: G06F 11/20 using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
• CPC Class Codes: G06F 11/2092 Techniques of failing over between control units
• CPC Class Codes: G06F 2201/835 Timestamp
• CPC Class Codes: G06F 3/0617 in relation to availability
• CPC Class Codes: G06F 3/0665 at area level, e.g. provisioning of virtual or logical volumes
• CPC Class Codes: G06F 3/0688 Non-volatile semiconductor memory arrays

Abstract

Techniques to achieve high availability in a multi-controller system are disclosed. In various embodiments, an indication is received that a network ping-based heartbeat mechanism has failed. In response to said indication, an alternative heartbeat is provided at least in part by periodically updating a heartbeat data stored in an on-disk heartbeat data structure, in a portion of a disk that is available to both an active controller and a standby controller of the multi-controller system.

Claims

1. A multi-controller system, comprising:

a processor configured to;

receive an indication that a network ping-based heartbeat mechanism has failed; and

in response to said indication, providing an alternative heartbeat at least in part by periodically updating a heartbeat data stored in an on-disk heartbeat data structure, in a portion of a disk that is available to both an active controller and a standby controller of the multi-controller system; and

a memory coupled to the processor and configured to provide the processor with instructions;

wherein the processor is further configured to detect that the on-disk heartbeat data structure-based alternative heartbeat has failed and in response switch over to a SCSI key refresh-based heartbeat mechanism.

2. The system of claim 1, wherein the standby controller is configured to read the updated heartbeat data from the on-disk heartbeat data structure.

3. The system of claim 2, wherein the standby controller is configured to read the heartbeat data from the on-disk heartbeat data structure;

determine whether the heartbeat data read from the on-disk heartbeat data structure has been updated within a prescribed interval; and

conclude based at least in part on the determination whether the active controller remains alive.

4. The system of claim 3, wherein the standby controller is further configured to update a standby controller portion of the on-disk heartbeat data structure, based at least in part on a determination that the heartbeat data read by the standby controller from the on-disk heartbeat data structure has been updated within the prescribed interval.

5. The system of claim 4, wherein the active controller is configured to read the standby controller portion of the on-disk heartbeat data structure;

determine based at least in part on the data read from the whether the standby controller portion of the on-disk heartbeat data structure has been updated within a prescribed interval; and

conclude based at least in part on the determination whether the standby controller is using the on-disk heartbeat data structure to determine whether the active controller is still alive.

6. The system of claim 1, wherein the on-disk heartbeat data structure comprises an identical data structure stored on each of a plurality of storage disks comprising the multi-controller system.

7. The system of claim 1, wherein the standby controller is configured to check to determine whether a key refresh has been performed by the active controller, and to determine based at least in part on a detected key refresh that the active controller is still alive.

8. The system of claim 7, wherein the standby controller concludes upon detecting a first refreshed key, without checking one or more other keys, that the active controller is still alive.

9. The system of claim 1, wherein the active controller is configured to refresh the respective SCSI key of a subset of disks comprising a storage array with which the multi-controller system is associated.

10. A method to achieve high availability in a multi-controller system, comprising:

receiving an indication that a network ping-based heartbeat mechanism has failed;

in response to said indication, providing an alternative heartbeat at least in part by periodically updating a heartbeat data stored in an on-disk heartbeat data structure, in a portion of a disk that is available to both an active controller and a standby controller of the multi-controller system; and

in response to detecting that the on-disk heartbeat data structure-based alternative heartbeat has failed, switching over to a SCSI key refresh-based heartbeat mechanism.

11. The method of claim 10, wherein the standby controller is configured to read the updated heartbeat data from the on-disk heartbeat data structure.

12. The method of claim 11, wherein the standby controller is configured to read the heartbeat data from the on-disk heartbeat data structure;

determine whether the heartbeat data read from the on-disk heartbeat data structure has been updated within a prescribed interval; and

conclude based at least in part on the determination whether the active controller remains alive.

13. The method of claim 12, wherein the standby controller is further configured to update a standby controller portion of the on-disk heartbeat data structure, based at least in part on a determination that the heartbeat data read by the standby controller from the on-disk heartbeat data structure has been updated within the prescribed interval.

14. The method of claim 13, wherein the active controller is configured to read the standby controller portion of the on-disk heartbeat data structure;

determine based at least in part on the data read from the whether the standby controller portion of the on-disk heartbeat data structure has been updated within a prescribed interval; and

conclude based at least in part on the determination whether the standby controller is using the on-disk heartbeat data structure to determine whether the active controller is still alive.

15. The method of claim 10, wherein the on-disk heartbeat data structure comprises an identical data structure stored on each of a plurality of storage disks comprising the multi-controller system.

16. The method of claim 10, wherein the standby controller is configured to check to determine whether a key refresh has been performed by the active controller, and to determine based at least in part on a detected key refresh that the active controller is still alive.

17. The method of claim 16, wherein the standby controller concludes upon detecting a first refreshed key, without checking one or more other keys, that the active controller is still alive.

18. A computer program product to achieve high availability in a multi-controller system, the computer program product being embodied in a non-transitory computer readable medium and comprising computer instructions for:

receiving an indication that a network ping-based heartbeat mechanism has failed;

in response to said indication, providing an alternative heartbeat at least in part by periodically updating a heartbeat data stored in an on-disk heartbeat data structure, in a portion of a disk that is available to both an active controller and a standby controller of the multi-controller system; and

in response to detecting that the on-disk heartbeat data structure-based alternative heartbeat has failed, switching over to a SCSI key refresh-based heartbeat mechanism.

19. The computer program product of claim 18, wherein the standby controller is configured to read the updated heartbeat data from the on-disk heartbeat data structure.

20. The computer program product of claim 19, wherein the standby controller is configured to read the heartbeat data from the on-disk heartbeat data structure;

determine whether the heartbeat data read from the on-disk heartbeat data structure has been updated within a prescribed interval; and

conclude based at least in part on the determination whether the active controller remains alive.