Adaptive scanning rates

• US 10,498,822 B2
• Filed: 12/05/2017
• Issued: 12/03/2019
• Est. Priority Date: 01/30/2015
• Status: Active Grant

• Current Assignee: Pure Storage, Inc.
• Original Assignee: Pure Storage, Inc.
• Inventors: Khadiwala, Ravi V., Wozniak, Ethan S., Resch, Jason K.
• Primary Examiner(s): Alphonse, Fritz
• Application Number:
  US15/831,965
• Publication Number:
  US 20180109614A1
• Time in Patent Office: 728 Days
• Field of Search: 714759
• CPC Class Codes: G06F 11/1076 Parity data used in redundant arrays of independent storages, e.g. in RAID systems
• CPC Class Codes: H03M 13/09 Error detection only, e.g. using cyclic redundancy check [CRC] codes or single parity bit
• CPC Class Codes: H03M 13/1515 Reed-Solomon codes
• CPC Class Codes: H03M 13/3761 using code combining, i.e. using combining of codeword portions which may have been transmitted separately, e.g. Digital Fountain codes, Raptor codes or Luby Transform [LT] codes
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

A distributed storage network (DSN) stores encoded data slices in dispersed storage (DS) memories using a DSN namespace divided into a plurality of DSN address ranges. Multiple scanning modules, each including a processor and associated memory, identify DSN address ranges for detection of storage errors within the DSN namespace, and initiate scanning for storage errors within the identified DSN address ranges. The scanning is performed in accordance with a scanning rate associated with each individual scanning module. Each scanning module obtains a target scanning performance level, which specifies a scanning performance level for the detection of storage errors across multiple DSN address ranges associated with more than one scanning module, and determines an actual scanning performance level for storage error detection across the same range of DSN addresses. The scanning modules cooperatively determine an updated scanning rate for each scanning module based on the target scanning performance level and the actual scanning performance level. The scanning rate of at least one of the plurality of scanning modules is altered to match the updated scanning rate.

Claims

1. A method for use in a distributed storage network (DSN) storing encoded data slices in dispersed storage (DS) memories using a DSN namespace divided into a plurality of DSN address ranges, the method comprising:

identifying, using a processor and associated memory included in each scanning module of a plurality of scanning modules, identified DSN address ranges for detection of storage errors within the DSN namespace;

initiating, using the processor and associated memory included in each scanning module of a plurality of scanning modules, scanning for the storage errors within the identified DSN address ranges, wherein each individual scanning module scans for errors in accordance with a scanning rate associated with the individual scanning module;

obtaining, using the processor and associated memory included in each scanning module, a target scanning performance level, the target scanning performance level specifying a scanning performance level for the detection of storage errors across DSN address ranges associated with more than one scanning module;

determining, using the processor and associated memory included in each scanning module, an actual scanning performance level for storage error detection across the plurality of DSN address ranges;

cooperatively determining, by the plurality of scanning modules, an updated scanning rate for each scanning module, the cooperatively determining based on the target scanning performance level and the actual scanning performance level; and

altering a scanning rate of at least one of the plurality of scanning modules to match the updated scanning rate.

2. The method of claim 1, further comprising:

throttling the scanning rate of the at least one of the plurality of scanning modules to less than maximum capacity in response to the updated scanning rate for the at least one of the plurality of scanning modules decreasing.

3. The method of claim 1, the cooperatively determining including:

determining, by each of scanning module, a scanning rate required among the plurality of scanning modules for the plurality of scanning modules to complete a scan of the DSN namespace within a given period of time.

4. The method of claim 1, further comprising:

in response to a determination, by an individual scanning module, that the individual scanning module is unable to scan at a rate sufficient to meet the updated scanning rate, transmitting a warning indicator from the individual scanning module to other scanning modules of a plurality of scanning modules.

5. The method of claim 1, further comprising:

in response to detecting a failure of an individual scanning module, transferring responsibility for scanning a DSN address range associated with the individual scanning module to one or more other storage modules.

6. The method of claim 1, wherein scanning for storage errors by an individual scanning module includes:

issuing a list slice request from the individual scanning module to a storage unit within a DSN address range associated with the individual scanning module; and

interpreting a response to the list slice request to identify a storage error.

7. The method of claim 1, wherein:

at least one scanning module is implemented by a storage unit including a processing core and a plurality of DS memories.

8. A scanning module for use in a distributed storage network (DSN) storing encoded data slices in dispersed storage (DS) memories using a DSN namespace divided into a plurality of DSN address ranges, the scanning module comprising:

a processor and associated memory;

a network interface coupled to the processor and associated memory, the network interface configured to couple the scanning module to a plurality of other scanning modules associated with different DSN address ranges for detection of storage errors within the DSN namespace;

the processor and associated memory configured to;

determine an identified DSN address range for detection of storage errors within the DSN namespace;

initiate scanning for the storage errors within the identified DSN address range, wherein scanning for the storage errors is performed in accordance with a scanning rate associated with the scanning module;

obtain a target scanning performance level, the target scanning performance level specifying a scanning performance level for the detection of storage errors across the DSN namespace;

determine an actual scanning performance level for storage error detection across the DSN namespace;

determine, in cooperation with the plurality of other scanning modules, an updated scanning rate for both the scanning module and the plurality of other scanning modules, based on the target scanning performance level and the actual scanning performance level; and

alter a scanning rate of the scanning module to match the updated scanning rate.

9. The scanning module of claim 8, the processor and associated memory further configured to:

throttle the scanning rate of the scanning module to less than maximum capacity in response to the updated scanning rate decreasing.

10. The scanning module of claim 8, the processor and associated memory further configured to:

determine a scanning rate required, among both the scanning module and the plurality of other scanning modules, to complete a scan of the DSN namespace within a given period of time.

11. The scanning module of claim 8, the processor and associated memory further configured to:

determine that the scanning module is unable to scan at a rate sufficient to meet the updated scanning rate; and

in response to determining that the scanning module is unable to scan at a rate sufficient to meet the updated scanning rate, transmit a warning indicator to the plurality of other scanning modules.

12. The scanning module of claim 8, the processor and associated memory further configured to:

detect a failure of at least one of the other scanning modules; and

in response to detecting the failure, transfer responsibility for scanning at least a portion of the identified DSN address range to one or more of the other storage modules.

13. The scanning module of claim 8, wherein scanning for storage errors by an individual scanning module includes:

issuing a list slice request from the individual scanning module to a storage unit within a DSN address range associated with the individual scanning module; and

interpreting a response to the list slice request to identify a storage error.

14. The scanning module of claim 8, wherein:

the scanning module is implemented by a storage unit including a processing core and a plurality of DS memories.

15. A distributed storage network (DSN), comprising:

a plurality of storage units, each of the plurality of storage units including a plurality of dispersed storage (DS) memories used to store encoded data slices in accordance with a DSN namespace divided into a plurality of DSN address ranges;

a plurality of scanning modules, each scanning module including;

a processor and associated memory;

a network interface coupled to the processor and associated memory, the network interface configured to couple the scanning module to a plurality of other scanning modules associated with different DSN address ranges for detection of storage errors within the DSN namespace;

the processor and associated memory configured to;

determine an identified DSN address range for detection of storage errors within the DSN namespace;

initiate scanning for the storage errors within the identified DSN address range, wherein scanning for the storage errors is performed in accordance with a scanning rate associated with the scanning module;

obtain a target scanning performance level, the target scanning performance level specifying a scanning performance level for the detection of storage errors across the DSN namespace;

determine an actual scanning performance level for storage error detection across the DSN namespace;

determine, in cooperation with the plurality of other scanning modules, an updated scanning rate for both the scanning module and the plurality of other scanning modules, based on the target scanning performance level and the actual scanning performance level; and

alter a current scanning rate of the scanning module to match an updated scanning rate.

16. The distributed storage network (DSN) of claim 15, the processor and associated memory further configured to:

throttle the scanning rate to less than maximum capacity in response to the updated scanning rate for the at least one of the plurality of scanning modules decreasing.

17. The distributed storage network (DSN) of claim 15, the processor and associated memory further configured to:

determine a scanning rate required, among both the scanning module and the plurality of other scanning modules, to complete a scan of the DSN namespace within a given period of time.

18. The distributed storage network (DSN) of claim 15, the processor and associated memory further configured to:

determine that the scanning module is unable to scan at a rate sufficient to meet the updated scanning rate; and

in response to determining that the scanning module is unable to scan at a rate sufficient to meet the updated scanning rate, transmit a warning indicator to the plurality of other scanning modules.

19. The distributed storage network (DSN) of claim 15, the processor and associated memory further configured to:

detect a failure of at least one of the other scanning modules; and

in response to detecting the failure, transfer responsibility for scanning at least a portion of the identified DSN address range to one or more of the other storage modules.

20. The distributed storage network (DSN) of claim 15, wherein scanning for storage errors by an individual scanning module includes:

issuing a list slice request from the individual scanning module to a storage unit within a DSN address range associated with the individual scanning module; and

interpreting a response to the list slice request to identify a storage error.