Adjusting dispersed storage network traffic due to rebuilding

• US 10,108,493 B2
• Filed: 08/22/2016
• Issued: 10/23/2018
• Est. Priority Date: 05/30/2013
• Status: Active Grant

• Current Assignee: Pure Storage, Inc.
• Original Assignee: International Business Machines Corporation
• Inventors: Volvovski, Ilya, Gladwin, S. Christopher, Grube, Gary W., Markison, Timothy W., Resch, Jason K., Shirley, Jr., Thomas Franklin, Dhuse, Greg, Motwani, Manish, Baptist, Andrew, Leggette, Wesley
• Primary Examiner(s): Ehne, Charles
• Application Number:
  US15/242,858
• Publication Number:
  US 20160357637A1
• Time in Patent Office: 792 Days
• CPC Class Codes: G06F 11/07 Responding to the occurrence of a fault, e.g. fault tolerance
• CPC Class Codes: G06F 11/1092 Rebuilding, e.g. when physically replacing a failing disk
• CPC Class Codes: G06F 11/1453 using de-duplication of the data
• CPC Class Codes: G06F 11/1456 Hardware arrangements for backup
• CPC Class Codes: G06F 11/1458 Management of the backup or restore process
• CPC Class Codes: G06F 11/2094 Redundant storage or storage space G06F11/2056 takes precedence
• CPC Class Codes: G06F 11/3034 where the computing system component is a storage system, e.g. DASD based or network based digital input from or digital output to record carriers G06F3/06; digital recording or reproducing G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097
• CPC Class Codes: G06F 11/3419 by assessing time
• CPC Class Codes: G06F 11/3433 for load management allocation of a server based on load conditions G06F9/505; load rebalancing G06F9/5083; redistributing the load in a network by a load balancer H04L67/1029
• CPC Class Codes: G06F 11/3485 for I/O devices
• CPC Class Codes: G06F 11/349 for interfaces, buses
• CPC Class Codes: G06F 2211/1028 Distributed, i.e. distributed RAID systems with parity
• CPC Class Codes: G06F 3/06 Digital input from, or digital output to, record carriers , e.g. RAID, emulated record carriers or networked record carriers
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/064 Management of blocks
• CPC Class Codes: G06F 3/0659 Command handling arrangements, e.g. command buffers, queues, command scheduling
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46

Abstract

A method includes identifying an encoded data slice for rebuilding. The method further includes determining whether a rebuilding threshold for the set of encoded data slices has been reached. When the rebuilding threshold has been reached, the method further includes determining, based on a condition of the DSN, whether to execute a rebuilding function, to delay execution of the rebuilding function, to adjust a rebuilding network protocol, or to modifying rebuilding criteria. When the determination is to execute the rebuilding function, the method further includes rebuilding the encoded data slice.

Claims

1. A method for execution by a device of a dispersed storage network (DSN), the method comprises:

identifying an encoded data slice for rebuilding, wherein a data segment of data is dispersed storage error encoded to produce a set of encoded data slices, which originally included the encoded data slice for rebuilding;

determining whether a rebuilding threshold for the set of encoded data slices has been reached;

when the rebuilding threshold has been reached, determining, based on a condition of the DSN, whether to execute a rebuilding function, to delay execution of the rebuilding function, to adjust a rebuilding network protocol, or to modifying rebuilding criteria; and

when the determination is to execute the rebuilding function, rebuilding the encoded data slice.

2. The method of claim 1, wherein the device comprises one or more of:

a computing device;

a storage unit;

a processing module within the computing device;

a processing module within the storage unit;

an integrity unit; and

a processing module within the integrity unit.

3. The method of claim 1, wherein the determining whether the rebuilding threshold has been reached comprises:

incrementing a rebuild count based on the identifying of the encoded data slice;

subtracting the rebuild count from a total number of encoded data slices in the set of encoded data slices to object a good encoded data slice count; and

when the good encoded data slice count is equal to or less than the rebuilding threshold, determining that the rebuilding threshold has been reached.

4. The method of claim 1 further comprises:

when the determination is to delay execution of the rebuilding function;

determining, based the condition of the DSN, a delay period;

queuing the rebuilding of the encoded data slice; and

when the delay period expires;

determining whether the condition of the DSN has been resolved; and

when the condition of the DSN has been resolved, executing the rebuilding of the encoded data slice.

5. The method of claim 4 further comprises:

when the condition of the DSN has not been resolved;

determining whether a rebuild count has increase since the encoded data slices was identified for rebuilding;

when the rebuild count has increased, comparing a rebuild urgency to the condition of the DSN, wherein the rebuild urgency increases proportionally with the rebuild count;

when the rebuild urgency compares unfavorably to the condition of the DSN, executing the rebuilding of the encoded data slice; and

when the rebuild urgency compares favorably to the condition of the DSN, extending the delay period.

6. The method of claim 1 further comprises:

when the determination is to adjust the rebuilding network protocol, implementing one or more of;

bundling rebuilding requests to a storage unit for multiple encoded data slices that have been identified for rebuilding;

serializing transmission of a set of rebuilding requests to storage units regarding the multiple encoded data slices that have been identified for rebuilding; and

assigning an individual response timeslot to each of multiple storage units for individually responding to a rebuild request regarding the multiple encoded data slices that have been identified for rebuilding.

7. The method of claim 1 further comprises:

when the determination is to modifying rebuilding criteria, implementing one or more of;

decreasing the rebuilding threshold;

increasing a wait time for a storage unit to come back on line, wherein the storage unit has encoded data slices identified for rebuilding; and

adjusting rebuild queue priorities of encoded data slices that have been identified for rebuilding.

8. The method of claim 1, wherein the condition of the DSN comprises one or more of:

latency in responding to data access requests;

volume of the data access requests;

latency in responding to system level access requests; and

volume of the system level access requests.

9. A device of a dispersed storage network (DSN), wherein the device comprises:

a network interface;

memory; and

a processing module operably coupled to the network interface and the memory, wherein the processing module is operable to;

identify an encoded data slice for rebuilding, wherein a data segment of data is dispersed storage error encoded to produce a set of encoded data slices, which originally included the encoded data slice for rebuilding;

determine whether a rebuilding threshold for the set of encoded data slices has been reached;

when the rebuilding threshold has been reached, determine, based on a condition of the DSN, whether to execute a rebuilding function, to delay execution of the rebuilding function, to adjust a rebuilding network protocol, or to modifying rebuilding criteria; and

when the determination is to execute the rebuilding function, rebuild the encoded data slice.

10. The device of claim 9, wherein the device comprises one or more of:

a computing device;

a storage unit; and

an integrity unit.

11. The device of claim 9, wherein the processing module further functions to determine whether the rebuilding threshold has been reached by:

incrementing a rebuild count based on the identifying of the encoded data slice;

subtracting the rebuild count from a total number of encoded data slices in the set of encoded data slices to object a good encoded data slice count; and

when the good encoded data slice count is equal to or less than the rebuilding threshold, determining that the rebuilding threshold has been reached.

12. The device of claim 9, wherein the processing module further functions to:

when the determination is to delay execution of the rebuilding function;

determine, based the condition of the DSN, a delay period;

queue the rebuilding of the encoded data slice; and

when the delay period expires;

determine whether the condition of the DSN has been resolved; and

when the condition of the DSN has been resolved, execute the rebuilding of the encoded data slice.

13. The device of claim 12, wherein the processing module further functions to:

when the condition of the DSN has not been resolved;

determine whether a rebuild count has increase since the encoded data slices was identified for rebuilding;

when the rebuild count has increased, compare a rebuild urgency to the condition of the DSN, wherein the rebuild urgency increases proportionally with the rebuild count;

when the rebuild urgency compares unfavorably to the condition of the DSN, execute the rebuilding of the encoded data slice; and

when the rebuild urgency compares favorably to the condition of the DSN, extend the delay period.

14. The device of claim 9, wherein the processing module further functions to:

when the determination is to adjust the rebuilding network protocol, implementing one or more of;

bundle rebuilding requests to a storage unit for multiple encoded data slices that have been identified for rebuilding;

serialize transmission of a set of rebuilding requests to storage units regarding the multiple encoded data slices that have been identified for rebuilding; and

assign an individual response timeslot to each of multiple storage units for individually responding to a rebuild request regarding the multiple encoded data slices that have been identified for rebuilding.

15. The device of claim 9, wherein the processing module further functions to:

when the determination is to modifying rebuilding criteria, implementing one or more of;

decrease the rebuilding threshold;

increase a wait time for a storage unit to come back on line, wherein the storage unit has encoded data slices identified for rebuilding; and

adjust rebuild queue priorities of encoded data slices that have been identified for rebuilding.

16. The device of claim 9, wherein the condition of the DSN comprises one or more of:

latency in responding to data access requests;

volume of the data access requests;

latency in responding to system level access requests; and

volume of the system level access requests.