System for rebuilding data in a dispersed storage network

• US 10,042,708 B2
• Filed: 10/21/2016
• Issued: 08/07/2018
• Est. Priority Date: 10/30/2015
• Status: Expired due to Fees

• Current Assignee: Pure Storage, Inc.
• Original Assignee: International Business Machines Corporation
• Inventors: Resch, Jason K.
• Primary Examiner(s): Chaudry, Mujtaba M
• Application Number:
  US15/331,214
• Publication Number:
  US 20170123918A1
• Time in Patent Office: 655 Days
• Field of Search: None
• CPC Class Codes: G06F 11/1092 Rebuilding, e.g. when physically replacing a failing disk
• CPC Class Codes: G06F 11/2094 Redundant storage or storage space G06F11/2056 takes precedence
• CPC Class Codes: G06F 2201/805 Real-time
• CPC Class Codes: G06F 3/0604 Improving or facilitating administration, e.g. storage management
• CPC Class Codes: G06F 3/061 Improving I/O performance
• CPC Class Codes: G06F 3/0611 in relation to response time
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/0635 by changing the path, e.g. traffic rerouting, path reconfiguration
• CPC Class Codes: G06F 3/064 Management of blocks
• CPC Class Codes: G06F 3/0659 Command handling arrangements, e.g. command buffers, queues, command scheduling
• CPC Class Codes: G06F 3/0665 at area level, e.g. provisioning of virtual or logical volumes
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: G06F 3/0689 Disk arrays, e.g. RAID, JBOD
• CPC Class Codes: H03M 13/1515 Reed-Solomon codes
• CPC Class Codes: H03M 13/3761 using code combining, i.e. using combining of codeword portions which may have been transmitted separately, e.g. Digital Fountain codes, Raptor codes or Luby Transform [LT] codes
• CPC Class Codes: H04L 43/0864 Round trip delays
• CPC Class Codes: H04L 43/0876 Network utilisation, e.g. volume of load or congestion level
• CPC Class Codes: H04L 43/16 Threshold monitoring
• CPC Class Codes: H04L 67/1008 based on parameters of servers, e.g. available memory or workload monitoring of computer activity G06F11/30
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

A method for execution by a dispersed storage and task (DST) execution unit includes receiving a request for rebuilding dispersed error encoded data slices and determining a rate for rebuilding the data slices. The method continues by the DST receiving one or more requests for dispersed error encoded data slices not associated with rebuilding and modifying the rate of processing for the one or more requests for dispersed error encoded data slices not associated with rebuilding dispersed error encoded data slices in accordance with the rate for rebuilding dispersed error encoded data slices. When the sending rebuilding dispersed error encoded data slices has concluded the method continues by reverting the rate of processing of requests for dispersed error encoded data slices to the previous rate.

Claims

1. A computing device comprising:

an interface configured to interface and communicate with a dispersed storage network (DSN);

memory that stores operational instructions; and

a processing module operably coupled to the interface and to the memory, wherein the processing module, when operable within the computing device based on the operational instructions is configured to;

transmit, via the interface, a rebuild request to at least a threshold number of storage units (SUs) that store a set of encoded data slices (EDSs) associated with a data object, wherein the data object is segmented into a plurality of data segments, and wherein a data segment of the plurality of data segments is dispersed error encoded in accordance with dispersed error encoding parameters to produce the set of EDSs;

receive at least a decode threshold number of EDSs from the at least a decode threshold number of SUs of the threshold number of SUs;

transmit, via the interface, an instruction to the DSN devices that are not recipients of the decode threshold number of EDSs to reduce a rate of processing of encoded data slice requests from a first rate to a second rate;

recover a rebuilt data segment based on the at least a decode threshold number of EDSs;

dispersed error encode the rebuilt data segment to generate a rebuilt set of EDSs;

transmit, via the interface, an EDS of the rebuilt set of EDSs to a storage unit (SU) of the threshold number of SUs from which none of the at least the decode threshold number of EDSs was received; and

transmit, via the interface, an instruction to the DSN devices that are not recipients of the decode threshold number of EDSs to increase the rate of processing of encoded data slice requests from the second rate to approximately the first rate.

2. The computing device of claim 1, wherein the processing module, when operable within the computing device based on the operational instructions is further configured to:

determine the second rate of processing of the encoded data slice requests based on at least one of responding to a read slice request from a dispersed storage network (DSN) integrity processing unit and generating a rebuilt encoded data slice.

3. The computing device of claim 1, wherein the processing module, when operable within the computing device based on the operational instructions, is further configured to:

determine the second rate of processing of the encoded data slice requests based on at least one of the rate of receiving one or more requests for the rebuilding slices, a predetermination, and a received rebuilding rate indicator.

4. The computing device of claim 1, wherein the processing module, when operable within the computing device based on the operational instructions, is further configured to:

determine the second rate of processing of encoded data slice requests based on one or more SUs issuing a rebuild request to a DSN integrity processing unit in response to receiving one or more requests for rebuilding dispersed error encoded data slices.

5. The computing device of claim 1, wherein the processing module, when operable within the computing device based on the operational instructions, is further configured to:

determine the second rate of processing of encoded data slice requests based on detecting the transmitting of the rebuilding slices based on at least one of responding to a read slice request, generating a rebuilt encoded data slice, receiving the requests for the rebuilding slices, a predetermination, and a received rebuilding rate indicator.

6. The computing device of claim 1, wherein the processing module, when operable within the computing device based on the operational instructions, is further configured to:

adjust the rate of processing of encoded data access requests such that one or more EDSs transmitted by the DSN are processed at a second rate by a particular SU of the at least a threshold number of SUs when the SU is simultaneously busy responding to a rebuild request.

7. The computing device of claim 1, wherein the processing module, when operable within the computing device based on the operational instructions, is further configured to:

transmit an instruction to the devices that are not recipients of the decode threshold number of EDSs to increase the rate of processing of encoded data slice requests from the second rate by an amount equal to the second rate times a multiplier.

8. The computing device of claim 1, wherein the processing module, when operable within the computing device based on the operational instructions, is further configured to:

transmit an instruction to the devices that are not recipients of the decode threshold number of EDSs to decrease the rate of processing of encoded data slice requests from the first rate to the second rate such that the second rate is equal to the first rate minus N.

9. The computing device of claim 1, wherein the processing module, when operable within the computing device based on the operational instructions, is further configured to:

transmit an instruction to the devices that are not recipients of the decode threshold number of EDSs to decrease the rate of processing of encoded data slice requests from the first rate in proportion to the second rate.

10. A first computing device comprising:

an interface configured to interface and communicate with a dispersed storage network (DSN);

memory that stores operational instructions; and

a processing module operably coupled to the interface and to the memory, wherein the processing module, when operable within the computing device based on the operational instructions, is configured to;

receive, from a second computing device, a rebuild request to rebuild a set of encoded data slices (EDSs) associated with a data object, wherein the data object is segmented into a plurality of data segments, and wherein a data segment of the plurality of data segments is dispersed error encoded in accordance with dispersed error encoding parameters to produce the set of EDSs;

determine whether the first computing device includes an EDS of the set of EDSs; and

when it is determined that the first computing device includes the EDS of the set of EDSs;

transmit, via the interface, the EDS of the set of EDSs to the second computing device and modify a rate by which the first computing device services at least one of a write request or read request from a first rate to a second rate during the rebuilding of the set of encoded data slices (EDSs) associated with a data object; and

modify the rate by which the first computing device services at least one of the write request or the read request from approximately the second rate to approximately the first rate after the rebuilding of the set of encoded data slices (EDSs) associated with a data object.

11. The first computing device of claim 10, wherein the processing module, when operable within the first computing device based on the operational instructions is further configured to:

modify a rate by which the first computing device services at least one of a write request or read request from a first rate to a second rate during the rebuilding of the set of encoded data slices (EDSs) associated with a data object based at least one of responding to a read slice request from a dispersed storage network (DSN) integrity processing unit and generating a rebuilt encoded data slice.

12. The first computing device of claim 10, wherein the processing module, when operable within the first computing device based on the operational instructions, is further configured to:

modify a rate by which the first computing device services at least one of a write request or read request from a first rate to a second rate during the rebuilding of the set of encoded data slices (EDSs) associated with a data object based on at least one of the rate of receiving one or more requests for the rebuilding slices, a predetermination, and a received rebuilding rate indicator.

13. The first computing device of claim 10, wherein the processing module, when operable within the first computing device based on the operational instructions, is further configured to:

modify a rate by which the first computing device services at least one of a write request or read request from a first rate to a second rate during the rebuilding of the set of encoded data slices (EDSs) associated with a data object based on the computing device responding to receiving one or more requests for rebuilding dispersed error encoded data slices after receiving a rebuild request from a DSN integrity processing unit.

14. The first computing device of claim 10, wherein the processing module, when operable within the first computing device based on the operational instructions, is further configured to:

modify a rate by which the first computing device services at least one of a write request or read request from a first rate to a second rate during the rebuilding of the set of encoded data slices (EDSs) associated with a data object based on detecting the transmitting of the rebuilding slices based on at least one of responding to a read slice request, generating a rebuilt encoded data slice, receiving the requests for the rebuilding slices, a predetermination, and a received rebuilding rate indicator.

15. The first computing device of claim 10, wherein the processing module, when operable within the first computing device based on the operational instructions, is further configured to:

adjust the rate of processing of encoded data access requests such that one or more EDSs transmitted by the first computing device are processed at a second rate when it is simultaneously busy responding to a rebuild request.

16. The first computing device of claim 10, wherein the processing module, when operable within the first computing device based on the operational instructions, is further configured to:

receiving an instruction to increase the rate of processing of encoded data slice requests from the second rate by an amount equal to the second rate times a multiplier.

17. The first computing device of claim 10, wherein the processing module, when operable within the first computing device based on the operational instructions, is further configured to:

receive an instruction to decrease the rate of processing of encoded data slice requests from the first rate to the second rate such that the second rate is equal to the first rate minus N.

18. The first computing device of claim 10, wherein the processing module, when operable within the first computing device based on the operational instructions, is further configured to:

receiving an instruction to decrease the rate of processing of encoded data slice requests from the first rate in proportion to the rebuilding rate.

19. A non-transitory computer readable storage medium comprises:

at least one memory section that stores operational instructions that, when executed by a processing system of a dispersed storage network (DSN) that includes a processor and a memory, causes the processing system to;

transmit a rebuild request to at least a threshold number of storage units (SUs) that store a set of encoded data slices (EDSs) associated with a data object, wherein the data object is segmented into a plurality of data segments, and wherein a data segment of the plurality of data segments is dispersed error encoded in accordance with dispersed error encoding parameters to produce the set of EDSs;

receive at least a decode threshold number of EDSs from the at least a decode threshold number of the threshold number of SUs;

transmit an instruction to those devices that are not recipients of the decode threshold number of EDSs to reduce the rate of processing of encoded data slice requests from a first rate to a second rate;

recover a rebuilt data segment based on the at least a decode threshold number of EDSs;

dispersed error encode the rebuilt data segment to generate a rebuilt set of EDSs;

transmit an EDS of the rebuilt set of EDSs to a storage unit (SU) of the threshold number of SUs from which none of the at least the decode threshold number of EDSs was received; and

transmit an instruction to the devices that are not recipients of the decode threshold number of EDSs to increase the rate of processing of encoded data slice requests from the second rate to approximately the first rate.