Distributed management of a storage system

• US 10,348,675 B1
• Filed: 09/01/2016
• Issued: 07/09/2019
• Est. Priority Date: 07/24/2014
• Status: Active Grant

• Current Assignee: Pure Storage, Inc.
• Original Assignee: Pure Storage, Inc.
• Inventors: Engel, Joern, Driscoll, Alan, Vachharajani, Neil, Karr, Ronald S.
• Primary Examiner(s): Thammavong, Prasith
• Assistant Examiner(s): Olson, Alex G
• Application Number:
  US15/254,293
• Time in Patent Office: 1,041 Days
• CPC Class Codes: G06F 12/145 the protection being virtual, e.g. for virtual blocks or segments before a translation mechanism
• CPC Class Codes: H04L 61/302 Administrative registration, e.g. for domain names at internet corporation for assigned names and numbers [ICANN]
• CPC Class Codes: H04L 61/5046 Resolving address allocation conflicts; Testing of addresses testing when self-assigning an address H04L61/5092
• CPC Class Codes: H04L 61/5053 Lease time; Renewal aspects
• CPC Class Codes: H04L 61/5084 Providing for device mobility network addressing or numbering for mobility support in wireless networks H04W8/26; mobile IP H04W80/04
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 69/28 Timers or timing mechanisms used in protocols

Abstract

Systems, apparatuses, methods, and computer-readable storage mediums for performing lease-based fencing using a time-limited lease window. During the time-limited lease window, writes to a shared storage medium are permitted, while writes are denied for expired leases. When a successful heartbeat is generated for a primary storage controller, the lease window is extended for the primary storage controller from the time of a previous heartbeat. Accordingly, a prolonged stall between successive heartbeats by the primary storage controller will result in the newly extended lease being expired at the time it is granted. This scheme prevents a split brain scenario from occurring when a secondary storage controller takes over as the new primary storage controller in response to detecting the stall.

Claims

1. A storage system comprising:

a plurality of storage controllers configured to couple to one or more client computer systems via one or more data communications links; and

one or more storage devices coupled to the plurality of storage controllers;

wherein a first storage controller of the plurality of storage controllers is configured to;

generate heartbeats on intervals of a first period of time;

issue, for a first portion of the storage devices, a lease responsive to generating a current heartbeat;

determine if a lease is valid;

responsive to determining that the lease is valid, perform a state-changing operation on the first portion of the storage devices;

detect a plurality of pending state-changing operations targeting the first portion of the storage devices;

determine a maximum number of state-changing operations that can be performed without causing data loss due to a failure of a fencing mechanism; and

perform a first number of state-changing operations on the first portion of the storage devices responsive to determining the particular lease is valid, wherein the first number is less than or equal to the maximum number; and

wherein a second storage controller of the plurality of storage controllers is configured to;

generate heartbeats on intervals of a second period of time;

issue, for a second portion of the storage devices, a lease responsive to generating a current heartbeat;

determine if a lease is valid; and

responsive to determining that the lease is valid, perform a state-changing operation on the second portion of the storage devices; and

wherein the first storage controller and the second storage controller are configured to perform non-state-changing operations on both portions of the storage devices.

2. The storage system as recited in claim 1, wherein the first storage controller is further configured to prevent the state-changing operation from being performed on the first portion of the storage devices without a valid lease, and wherein the second storage controller does not have a valid lease on the first portion of the storage devices and the second storage controller cannot perform any state-changing operations on the first portion of the storage devices.

3. The storage system as recited in claim 2, wherein the first storage controller is further configured to:

recheck if the particular lease is valid responsive to performing the first number of state-changing operations on the first portion of the storage devices; and

perform a second number of state-changing operations on the first portion of the storage devices responsive to determining the particular lease is valid, wherein the second number is less than or equal to the maximum number.

4. The storage system as recited in claim 1, wherein the first storage controller is a primary storage controller for the first portion of the storage devices, and wherein a second storage controller of the plurality of storage controllers is configured to take over as a new primary storage controller for the first portion of the storage devices responsive to detecting that the first storage controller has not generated a heartbeat for a particular period of time.

5. The storage system as recited in claim 4, wherein responsive to taking over as the new primary storage controller for the first portion of the storage devices, the second storage controller is configured to:

generate heartbeats on intervals of the first period of time;

issue a new lease for the first portion of the storage devices responsive to generating a heartbeat, wherein a beginning of the new lease is calculated from a prior heartbeat and has a duration of a third period of time; and

determine if a particular lease is valid prior to performing a state-changing operation on the first portion of the storage devices.

6. The storage system as recited in claim 1, wherein the second storage controller is further configured to prevent state-changing operations from being performed on the second portion of the storage devices without a valid lease, and wherein the first storage controller does not have a valid lease on the second portion of the storage devices and the first storage controller cannot perform any state-changing operations on the second portion of the storage devices.

7. The storage system as recited in claim 1, further comprising:

sending, from the first storage controller to the second storage controller, heartbeats via a data communications bus between the first storage controller and the second storage controller; and

sending, from the second storage controller to the first storage controller, heartbeats via a data communications bus between the first storage controller and the second storage controller.

8. A plurality of storage controllers:

wherein each storage controller is configured to couple to one or more client computer systems via one or more data communications links and each storage controller is coupled to a plurality of storage devices;

wherein a first storage controller is configured to;

generate heartbeats on intervals of a first period of time;

issue, for a first portion of the storage devices, a lease responsive to generating a current heartbeat;

determine if a lease is valid;

responsive to determining that the lease is valid, perform a state-changing operation on the first portion of the storage devices;

detect a plurality of pending state-changing operations targeting the first portion of the storage devices; and

determine a maximum number of state-changing operations that can be performed without causing data loss due to a failure of a fencing mechanism; and

perform a first number of state-changing operations on the first portion of the storage devices responsive to determining the particular lease is valid, wherein the first number is less than or equal to the maximum number; and

wherein a second storage controller is configured to;

generate heartbeats on intervals of a second period of time;

issue, for a second portion of the storage devices, a lease responsive to generating a current heartbeat;

determine if a lease is valid; and

responsive to determining that the lease is valid, perform a state-changing operation on the second portion of the storage devices; and

wherein the first storage controller and the second storage controller are configured to perform non-state-changing operations on both portions of the storage devices.

9. The plurality of storage controllers as recited in claim 8, wherein the first storage controller is further configured to prevent the state-changing operation from being performed on the first portion of the storage devices without a valid lease, and wherein the second storage controller does not have a valid lease on the first portion of the storage devices and the second storage controller cannot perform any state-changing operations on the first portion of the storage devices.

10. The plurality of storage controllers as recited in claim 9, wherein the first storage controller is further configured to:

recheck if the particular lease is valid responsive to performing the first number of state-changing operations on the first portion of the storage devices; and

perform a second number of state-changing operations on the first portion of the storage devices responsive to determining the particular lease is valid, wherein the second number is less than or equal to the maximum number.

11. The plurality of storage controllers as recited in claim 8, wherein the second storage controller is further configured to prevent state-changing operations from being performed on the second portion of the storage devices without a valid lease, and wherein the first storage controller does not have a valid lease on the second portion of the storage devices and the first storage controller cannot perform any state-changing operations on the second portion of the storage devices.

12. The plurality of storage controllers as recited in claim 8, wherein:

the first storage controller is configured to send, to the second storage controller, heartbeats via a data communications bus between the first storage controller and the second storage controller; and

the second storage controller is configured to send, to the first storage controller, heartbeats via a data communications bus between the first storage controller and the second storage controller.

13. A method comprising:

generating, by a first storage controller, heartbeats on intervals of a first period of time;

issuing, by the first storage controller for a first portion of the storage devices, a lease responsive to generating a current heartbeat;

determining, by the first storage controller, if a lease is valid;

responsive to determining that the lease is valid, perform by the first storage controller, a state-changing operation on the first portion of the storage devices;

generating, by a second storage controller, heartbeats on intervals of a second period of time;

issuing, by the second storage controller for a second portion of the storage devices, a lease responsive to generating a current heartbeat;

determining, by the second storage controller, if a lease is valid;

responsive to determining that the lease is valid, perform by the second storage controller, a state-changing operation on the second portion of the storage devices;

detecting, by the first storage controller, a plurality of pending state-changing operations targeting the first portion of the storage devices;

determining, by the first storage controller, a maximum number of state-changing operations that can be performed without causing data loss due to a failure of a fencing mechanism; and

performing, by the first storage controller, a first number of state-changing operations on the first portion of the storage devices responsive to determining the particular lease is valid, wherein the first number is less than or equal to the maximum number;

wherein the first storage controller and the second storage controller are configured to perform non-state-changing operations on both portions of the storage devices.

14. The method as recited in claim 13, further comprising preventing, by the first storage controller, the state-changing operation from being performed on the first portion of the storage devices without a valid lease, and wherein the second storage controller does not have a valid lease on the first portion of the storage devices and the second storage controller cannot perform any state-changing operations on the first portion of the storage devices.

15. The method as recited in claim 14, further comprising:

rechecking, by the first storage controller, if the particular lease is valid responsive to performing the first number of state-changing operations on the first portion of the storage devices; and

performing, by the first storage controller, a second number of state-changing operations on the first portion of the storage devices responsive to determining the particular lease is valid, wherein the second number is less than or equal to the maximum number.

16. The method as recited in claim 13, wherein the second storage controller is further configured to prevent state-changing operations from being performed on the second portion of the storage devices without a valid lease, and wherein the first storage controller does not have a valid lease on the second portion of the storage devices and the first storage controller cannot perform any state-changing operations on the second portion of the storage devices.

17. The method as recited in claim 13, further comprising:

sending, from the first storage controller to the second storage controller, heartbeats via a data communications bus between the first storage controller and the second storage controller; and

sending, from the second storage controller to the first storage controller, heartbeats via a data communications bus between the first storage controller and the second storage controller.