Multiple-apartment aware offlining of devices for disruptive and destructive operations

• US 10,705,732 B1
• Filed: 12/13/2017
• Issued: 07/07/2020
• Est. Priority Date: 12/08/2017
• Status: Active Grant

• Current Assignee: Pure Storage, Inc.
• Original Assignee: Pure Storage, Inc.
• Inventors: Bernat, Andrew R., Inozemtsev, Grigori, McLeary, Iris, Scholbrock, Benjamin
• Primary Examiner(s): Hur, J. H.
• Application Number:
  US15/840,810
• Time in Patent Office: 937 Days
• Field of Search: None
• CPC Class Codes: G06F 11/2089 Redundant storage control functionality
• CPC Class Codes: G06F 11/3055 Monitoring arrangements for monitoring the status of the computing system or of the computing system component, e.g. monitoring if the computing system is on, off, available, not available error or fault processing without redundancy G06F11/0703; error detection or correction by redundancy in data representation G06F11/08; error detection or correction by redundancy in operation G06F11/14; error detection or correction by redundancy in hardware G06F11/16
• CPC Class Codes: G06F 2201/80 Database-specific techniques
• CPC Class Codes: G06F 2201/88 Monitoring involving counting
• CPC Class Codes: G06F 3/0602 specifically adapted to achieve a particular effect
• CPC Class Codes: G06F 3/0607 by facilitating the process of upgrading existing storage systems, e.g. for improving compatibility between host and storage device
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/0659 Command handling arrangements, e.g. command buffers, queues, command scheduling
• CPC Class Codes: G06F 3/0662 Virtualisation aspects
• CPC Class Codes: G06F 3/0665 at area level, e.g. provisioning of virtual or logical volumes
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: G06F 3/0688 Non-volatile semiconductor memory arrays
• CPC Class Codes: G06F 3/0689 Disk arrays, e.g. RAID, JBOD
• CPC Class Codes: G06F 8/654 using techniques specially adapted for alterable solid state memories, e.g. for EEPROM or flash memories

Abstract

A method for taking a storage device offline, performed by a storage system, is provided. The method includes reserving the storage device and stopping direction to increment a heartbeat counter of the storage device. The method includes verifying the heartbeat counter of the storage device is static, and performing, responsive to the verifying, an operation on the storage device, with the storage device offline relative to the storage system I/O of user data.

Claims

1. A method comprising:

reserving a storage device, wherein the reservation causes a storage controller of a storage system to prevent other storage controllers of the storage system from accessing the storage device;

stopping sending direction to the storage device to increment a heartbeat counter of the storage device;

verifying the heartbeat counter of the storage device is static; and

performing, responsive to the verifying, an operation on the storage device, with the storage device offline relative to the storage system I/O of user data.

2. The method of claim 1, further comprising:

continuing the storage system I/O of user data to other storage devices of the plurality of storage devices of the storage system, with the storage device offline.

3. The method of claim 1, wherein:

the verifying is after a specified time interval from the stopping; and

the specified time interval is pre-defined for subsystems of the storage system, to ensure an incrementing of the heartbeat counter of the storage device by one of the subsystems of the storage system is detectable by the verifying after the specified time interval.

4. The method of claim 1, wherein:

each of the directions to the storage device that increment the heartbeat counter comprises a SCSI (small computer systems interface) PRO (persistent reserve out) command; and

the verifying that the heartbeat counter is static comprises a SCSI PRI (persistent reserve in) command to read the heartbeat counter.

5. The method of claim 1, wherein the performing the operation comprises reading one or more logs from the storage device.

6. The method of claim 1, wherein the storage system comprises two or more storage arrays or storage clusters each having one or more processors or controllers.

7. The method of claim 1, wherein the reserving the storage device comprises writing a status of the storage device as reserved to a list of statuses of the plurality of storage devices of the storage system.

8. A tangible, non-transitory, computer-readable media having instructions thereupon which, when executed by a processor, cause the processor to perform a method comprising:

reserving a storage device of a plurality of storage devices of a storage system, to take the storage device offline relative to storage system I/O (input/output) of user data while other storage devices of the plurality of storage devices remain online, wherein the reservation causes a storage controller of the storage system to prevent other storage controllers of the storage system from accessing the storage device;

ceasing sending of a command to the storage device that increments a heartbeat counter of the storage device;

determining, after a specified time interval, that the heartbeat counter of the storage device is static; and

performing an operation on the storage device, responsive to the determining, with the storage device offline.

9. The computer-readable media of claim 8, wherein the method further comprises:

sending the command that increments the heartbeat counter to the other storage devices of the plurality of storage devices that remain online, repeated at a further specified time interval that is less than or equal to the specified time interval for determining that the heartbeat counter of the offline storage device is static.

10. The computer-readable media of claim 8, wherein:

the command that increments the heartbeat counter comprises a SCSI (small computer systems interface) PRO (persistent reserve out) command; and

the determining that the heartbeat counter is static comprises sending a SCSI PRI (persistent reserve in) command to read the heartbeat counter.

11. The computer-readable media of claim 8, wherein the performing the operation comprises reading a log from the storage device.

12. The computer-readable media of claim 8, wherein the method further comprises:

communicating among two or more storage arrays or storage clusters, coupled together as the storage system and each having one or more controllers.

13. The computer-readable media of claim 8, wherein the reserving the one of the plurality of storage devices comprises writing to a list comprising a status of each of the plurality of storage devices that the status of the storage device is reserved.

14. A storage system, comprising:

a plurality of storage devices, each having a heartbeat counter; and

one or more processors, operatively coupled to the plurality of storage devices, to;

send a command to each of the plurality of storage devices that is online to increment the heartbeat counter;

verify incrementing of the heartbeat counter of each of the plurality of storage devices that is online;

reserve a storage device of the plurality of storage devices as offline relative to storage system I/O (input/output) of user data, wherein the reservation causes a storage controller of the storage system to prevent other storage controllers of the storage system from accessing the storage device;

cease sending the command to increment the heartbeat counter to the reserved storage device of the plurality of storage devices;

verify that the heartbeat counter of the reserved storage device is static; and

perform an operation on the reserved storage device, as offline, responsive to verifying the heartbeat counter of the reserved storage device is static.

15. The storage system of claim 14, wherein the one or more processors are further to:

perform storage system I/O of user data to online storage devices of the plurality of storage devices, with the storage device offline.

16. The storage system of claim 14, wherein to verify that the heartbeat counter of the reserved storage device is static, the one or more processors are further to read the heartbeat counter of the reserved storage device after a specified time interval after ceasing sending the command to increment the heartbeat counter.

17. The storage system of claim 14, wherein:

to send a command to increment the heartbeat counter, the one or more processors are to send a SCSI (small computer systems interface) PRO (persistent reserve out) command;

to verify the incrementing of the heartbeat counter, the one or more processors are to send a SCSI PRI (persistent reserve in) command to read the heartbeat counter; and

to verify the heartbeat counter is static, the one or more processors are to send a subsequent SCSI PRI.

18. The storage system of claim 14, wherein to perform the operation, the one or more processors are further to read one or more logs from the reserved storage device.

19. The storage system of claim 14, wherein the plurality of storage devices are arranged in two or more storage arrays or storage clusters coupled together as the storage system.

20. The storage system of claim 14, wherein to reserve the one of the plurality of storage devices as offline the one or more processors are further to write to a list of storage devices a status of each storage device of the storage system indicating that the status of the one of the plurality of storage devices is reserved.