Storage systems with peer data scrub

• US 10,649,843 B2
• Filed: 08/03/2018
• Issued: 05/12/2020
• Est. Priority Date: 08/03/2018
• Status: Active Grant

• Current Assignee: JP Morgan Chase Bank, N.A. (JP Morgan Chase & Co)
• Original Assignee: Western Digital Technologies Incorporated (Western Digital Corporation)
• Inventors: Roberts, Adam
• Primary Examiner(s): Tabone, Jr., John J
• Application Number:
  US16/054,940
• Publication Number:
  US 20200042380A1
• Time in Patent Office: 648 Days
• Field of Search: 714764, 714769, 714770, 714773, 714 623, 714 48, 714 52, 714 54, 714 57
• CPC Class Codes: G06F 11/0766 Error or fault reporting or storing
• CPC Class Codes: G06F 11/106 Correcting systematically all correctable errors, i.e. scrubbing
• CPC Class Codes: G06F 11/1092 Rebuilding, e.g. when physically replacing a failing disk
• CPC Class Codes: G06F 2211/1028 Distributed, i.e. distributed RAID systems with parity
• CPC Class Codes: G06F 2211/1088 Scrubbing in RAID systems with parity
• CPC Class Codes: G06F 3/0614 Improving the reliability of storage systems
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/0653 Monitoring storage devices or systems
• CPC Class Codes: G06F 3/0659 Command handling arrangements, e.g. command buffers, queues, command scheduling
• CPC Class Codes: G06F 3/0688 Non-volatile semiconductor memory arrays
• CPC Class Codes: G06F 3/0689 Disk arrays, e.g. RAID, JBOD
• CPC Class Codes: H04L 67/104 Peer-to-peer [P2P] networks
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

Example peer storage systems, storage devices, and methods provide data scrub using a peer communication channel. Peer storage devices establish peer communication channels that communicate data among the peer storage devices. A storage device may identify data segments from their storage medium for a data scrub process. A peer storage device may be identified that contains corresponding data segments to the data segment being scrubbed. A corresponding lock command may be sent over the peer communication channel to lock the corresponding data segments during the data scrub process. A data scrub error report may be generated from the data scrub process. If an error is discovered during the data scrub process the storage device may use the peer communication channel to retrieve recovery data from peer storage devices to rebuild the data segment with the error.

Claims

1. A storage device, comprising:

a processor;

a memory;

at least one storage medium;

an interface configured to communicate with a plurality of peer storage devices over a peer communication channel; and

a data scrub module stored in the memory and executable by the processor to perform operations comprising;

selecting a first data segment on the at least one storage medium for a data scrub process;

identifying at least one peer storage device including at least one corresponding data segment;

sending a corresponding data lock command to the at least one peer storage device over the peer communication channel;

performing a data scrub operation on the first data segment; and

generating a data scrub error report for the first data segment in response to performing the data scrub operation.

2. The storage device of claim 1, wherein the peer communication channel bypasses a storage control plane for managing the storage device.

3. The storage device of claim 1, further comprising:

a peer device registry stored in the memory and comprising;

a peer storage device identifier for each of the plurality of peer storage devices, wherein the peer storage device identifier is used to address data sent over the peer communication channel; and

a peer data configuration identifying corresponding data segments in the plurality of peer storage devices, wherein identifying at least one peer storage device including at least one corresponding data segment includes accessing the peer data configuration.

4. The storage device of claim 3, wherein:

the peer data configuration is a redundant array of independent disks (RAID) configuration;

the first data segment is a local portion of a RAID stripe; and

the at least one corresponding data segment includes at least one additional portion of the RAID stripe.

5. The storage device of claim 1, wherein the data scrub module is further executable by the processor to perform operations comprising:

storing a data scrub configuration in the memory, wherein the data scrub configuration defines a data scrub schedule and a data scrub pattern; and

initiating the data scrub process according to the data scrub schedule, wherein the first data segment is selected in response to the data scrub pattern.

6. The storage device of claim 5, wherein the data scrub module is further executable by the processor to perform operations comprising:

receiving the data scrub configuration from a data scrub manager, wherein the data scrub manager includes one of;

a peer data scrub manager instantiated in a manager peer storage device selected from the plurality of peer storage devices and communicating over the peer communication channel; and

a controller data scrub manager instantiated in a storage control plane for the plurality of peer storage devices.

7. The storage device of claim 6, wherein the data scrub process includes:

a first scrub portion performed by the data scrub manager for a first plurality of data segments on the at least one storage medium; and

a second scrub portion performed by the data scrub module for a second plurality of data segments on the at least one storage medium, wherein;

the second plurality of data segments includes the first data segment; and

the second scrub portion of the data scrub process is initiated in response to receiving the data scrub configuration from the data scrub manager.

8. The storage device of claim 1, wherein the data scrub module is further executable by the processor to perform operations comprising:

selecting a peer data segment on at least one peer storage medium for a peer data scrub process, wherein the peer storage medium is in a selected peer storage device from the plurality of peer storage devices;

performing a peer data scrub operation on the peer data segment over the peer communication channel; and

generating a peer data scrub error report for the peer data segment in response to performing the peer data scrub operation.

9. The storage device of claim 1, wherein the data scrub module is further executable by the processor to perform operations comprising:

monitoring a compute resource usage value related to the processor and the memory; and

scheduling the data scrub process in response to the compute resource usage value being below a compute resource availability threshold.

10. The storage device of claim 1, further comprising:

a self-healing module stored in the memory and executable by the processor to perform operations comprising;

identifying a failed data unit from the first data segment in response to the data scrub process;

receiving recovery data for the failed data unit from a recovery peer storage device over the peer communication channel; and

recovering the failed data unit from a first storage media segment in the at least one storage medium to a second storage media segment using the recovery data.

11. A computer-implemented method, comprising:

storing storage device identifiers for a plurality of peer storage devices in a first storage device;

establishing a peer communication channel with at least one of the plurality of peer storage devices from the first storage device, wherein the peer communication channel bypasses a storage control plane for managing the first storage device; and

executing a data scrub process comprising;

selecting a first data segment in the first storage device;

identifying at least one corresponding data segment in at least one corresponding peer storage device;

sending a corresponding data lock command to the at least one corresponding peer storage device over the peer communication channel;

performing a data scrub operation on the first data segment; and

generating a data scrub error report for the first data segment in response to performing the data scrub operation.

12. The computer-implemented method of claim 11, wherein identifying at least one corresponding data segment includes:

accessing a peer redundant array of independent disks (RAID) data configuration configured to identify corresponding data segments for the first data segment in the plurality of peer storage devices, wherein;

the first data segment is a local portion of a RAID stripe stored in the first storage device; and

the at least one corresponding data segment includes at least one additional portion of the RAID stripe.

13. The computer-implemented method of claim 11, further comprising:

storing a data scrub configuration, wherein the data scrub configuration defines a data scrub schedule and a data scrub pattern; and

initiating the data scrub process according to the data scrub schedule, wherein the first data segment is selected in response to the data scrub pattern.

14. The computer-implemented method of claim 13, further comprising:

receiving the data scrub configuration from a data scrub manager, wherein the data scrub manager includes one of;

a peer data scrub manager instantiated in a manager peer storage device selected from the plurality of peer storage devices and communicating over the peer communication channel; and

a controller data scrub manager instantiated in a storage control plane for the plurality of peer storage devices.

15. The computer-implemented method of claim 11, further comprising:

executing a peer data scrub process using the first storage device comprising;

selecting a peer data segment on at least one peer storage medium, wherein the peer storage medium is in a selected peer storage device from the plurality of peer storage devices and is not the first storage device;

performing a peer data scrub operation on the peer data segment over the peer communication channel; and

generating a peer data scrub error report for the peer data segment in response to performing the peer data scrub operation.

16. The computer-implemented method of claim 11, further comprising:

monitoring a compute resource usage value related to at least one compute resource of the first storage device; and

scheduling the data scrub process in response to the compute resource usage value being below a compute resource availability threshold.

17. The computer-implemented method of claim 11, further comprising:

identifying a failed data unit from the first data segment in response to the data scrub process;

receiving recovery data for the failed data unit from a recovery peer storage device over the peer communication channel; and

recovering the failed data unit from a first storage media segment in the first storage device to a second storage media segment using the recovery data.

18. A storage system, comprising:

a plurality of peer storage devices, wherein each of the plurality of peer storage devices comprises;

at least one storage medium;

means for storing storage device identifiers for the plurality of peer storage devices;

means for establishing a peer communication channel between a scrubbing peer storage device selected from the plurality of peer storage devices and a target peer storage device selected from the plurality of peer storage devices, wherein the peer communication channel bypasses a storage control plane for managing the plurality of peer storage devices;

means for the scrubbing peer storage device to select a first data segment in the target peer storage device;

means for identifying at least one corresponding data segment in at least one corresponding peer storage device;

means for sending a corresponding data lock command to the at least one corresponding peer storage device over the peer communication channel;

means for the scrubbing peer storage device to perform a data scrub operation on the first data segment over the peer communication channel; and

means for generating a data scrub error report for the first data segment in response to performing the data scrub operation.

19. The storage system of claim 18, further comprising:

means for managing a plurality of data scrub operations for the plurality of peer storage devices, wherein;

the means for managing the plurality of data scrub operations is configured to send a data scrub configuration to the plurality of peer storage devices; and

the data scrub configuration defines a data scrub schedule, a data scrub pattern, the scrubbing peer storage device, and the target peer storage device for each of the plurality of data scrub operations.

20. The storage system of claim 18, further comprising:

means for recovering at least one failed data unit from the first data segment using recovery data from the plurality of peer storage devices, wherein the means for recovering the at least one failed data unit is configured to;

automatically respond to the data scrub operation; and

transfer the recovery data between the peer storage devices over the peer communication channel.